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Accountability  *  Integrity  *  Reliability 


United  States  General  Accounting  Office 
Washington,  DC  20548 


July  31,  2001 

The  Honorable  Christopher  Shays 
Chairman,  Subcommittee  on  National  Security,  Veterans 
Affairs,  and  International  Relations 
Committee  on  Government  Reform 
House  of  Representatives 

Dear  Mr.  Chairman: 

The  amount  of  money  that  the  Department  of  Defense  (DOD)  spends  each 
year  contracting  for  goods  and  services  dwarfs  the  amounts  spent  by  other 
federal  agencies.  For  example,  during  fiscal  year  1999,  DOD  reported  that 
it  spent  about  $130  billion  on  contracts  for  goods  and  services.  By 
comparison,  the  second  largest  contractor  of  goods  and  services  in  the 
federal  government  was  the  Department  of  Energy,  which  reported  that  it 
spent  about  $15.5  billion  during  the  same  period. 

Since  1992,  we  have  designated  DOD’s  management  of  its  contracts  a  high- 
risk  area.  It  remains  on  our  list  of  high-risk  areas  today  because,  among 
other  things,  significant  challenges  still  face  DOD  in  overseeing  contracts 
and  preventing  erroneous  and  improper  payments  to  contractors.1 

With  the  goal  of  improving  its  ability  to  contract  for  goods  and  services, 
DOD  is  about  5  years  into  what  is  now  at  least  an  8-year  effort  to  acquire 
and  implement  a  Standard  Procurement  System  (SPS).  According  to  DOD, 
SPS  is  to  support  about  43,000  users  at  1,100  sites  worldwide  in  preparing, 
awarding,  and  administering  contracts.  SPS  also  is  to  interface  with  key 
financial  management  functions,  such  as  payment  processing.  Among 
other  things,  DOD  expects  SPS  to  replace  legacy  systems  that  support 
divergent  contracting  processes  and  procedures  across  component 
organizations  with  a  single,  standard  system  that  provides  electronic 
commerce  capabilities  and  uses  a  common  data  repository. 

As  agreed  with  your  office,  we  assessed  DOD’s  effectiveness  in  managing 
its  SPS  investment.  Our  objectives  were  to  determine  (1)  the  progress 
DOD  has  made  against  SPS  program  commitments  (i.e.,  cost,  schedule, 
and  performance  expectations,  including  delivery  of  promised  system 


1 High-Risk  Series:  An  Update  (GAO-Ol-263,  January  2001). 
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capabilities  and  expected  benefits)  and  (2)  whether  DOD  has 
economically  justified  further  investment  in  SPS.  Our  objectives,  scope, 
and  methodology  are  discussed  in  detail  in  appendix  I. 


Results  in  Brief 


DOD  has  not  met  key  SPS  program  commitments  and  does  not  know 
whether  it  is  meeting  others.  When  it  began  SPS  in  late  1994,  DOD  sought 
to  acquire  and  fully  implement  a  commercially  available  contract 
management  system  by  March  31,  2000.  At  that  time,  DOD  estimated  life- 
cycle  costs  of  the  system  to  be  approximately  $3  billion  over  a  10-year 
period.2  However,  full  implementation  has  been  delayed  3JA  years. 
According  to  the  SPS  program  manager,  this  delay  is  due  in  part  to  defined 
requirements  not  being  reflective  of  all  user  needs,  thereby  requiring  more 
time  and  resources  than  originally  planned  to  enhance  the  commercial 
product.3  The  commercial  product’s  software  required  extensive 
modification  to  meet  DOD’s  needs,  meaning  that  SPS  will  be  a  custom- 
developed  system  solution,  and  DOD  will  not  accrue  the  benefits  or 
reduced  cost  associated  with  using  a  commercial  product. 

Further,  for  software  releases  that  have  been  implemented,  DOD  has  not 
validated  that  each  release  is  producing  the  expected  benefits; 
consequently,  the  department  does  not  know  what  progress,  if  any,  has 
been  made  against  return-on-investment  goals.  Similarly,  DOD  is  not 
capturing  total  expenditures  and,  thus,  does  not  know  its  progress  against 
original  cost  projections,  which  have  since  grown  to  about  $3.7  billion. 
According  to  officials  in  the  SPS  program  office  and  the  Office  of  the 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications, 
and  Intelligence,4  who  are  responsible  for  managing,  overseeing,  and 
approving  DOD’s  investment  in  SPS,  they  are  not  measuring  progress 
because  there  is  no  departmental  requirement  to  do  so,  and,  thus,  it  is  not 
their  responsibility.  We  disagree.  Not  only  does  DOD  policy  require 
progress  on  programs  such  as  SPS  to  be  reported,  but  Clinger-Cohen  Act 
requirements  and  Office  of  Management  and  Budget  guidance  also  address 


2Life-cycle  costs  are  the  total  costs  to  the  government  of  the  system  over  its  useful  life. 
These  costs  include  acquisition,  operation,  maintenance,  and  disposal  of  a  system. 

According  to  the  SPS  Joint  Requirements  Board,  which  is  responsible  for  ensuring  the 
clarity  and  completeness  of  SPS  functional  and  interface  requirements,  the  original 
requirements  for  SPS  have  not  changed;  rather,  additional  clarification  was  provided  to 
better  ensure  that  user  needs  would  be  met. 

4This  office  is  DOD’s  Office  of  the  Chief  Information  Officer  (CIO). 
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the  need  to  measure  progress  to  ensure  that  programs  are  delivering  on 
commitments. 

DOD  also  has  not  economically  justified  its  investment  in  SPS.  DOD’s 
economic  justification  for  SPS  is  an  analysis  of  costs  and  benefits  that  was 
prepared  6  years  ago  and  updated  first  in  1997  and  again  in  2000.  However, 
these  analyses  are  not  credible,  and  both  updates  showed  that  SPS  was 
not  a  prudent  investment  because  the  system  would  have  a  negative  net 
present  value  (i.e.,  estimated  benefits  to  be  realized  would  not  exceed 
estimated  program  costs).  Specifically,  the  1997  economic  analysis 
estimated  life-cycle  costs  at  $2.9  billion  and  benefits  at  $1.8  billion,  while 
the  most  recent  analysis  showed  life-cycle  costs  to  be  $3.7  billion  and 
benefits  to  be  $1.4  billion.  Using  DOD’s  estimates,  we  calculated  SPS’  net 
present  value  to  be  about  negative  $174  million  for  fiscal  year  1997  and 
negative  $655  million  for  fiscal  year  2000.  Additionally,  since  these 
analyses  were  prepared,  system  capabilities  and  benefits  have  been 
deferred,  potentially  increasing  costs,  meaning  that  the  system’s  return  on 
investment  is  even  less  than  currently  predicted. 

Moreover,  DOD  has  not  divided  SPS  into  incremental  investment  decisions 
coinciding  with  incremental  release  implementations.  Instead,  DOD  has 
continued  to  treat  investment  in  SPS  as  an  “all  or  nothing”  decision.  Such  a 
monolithic  approach  to  investing  in  large  system  acquisitions,  like  SPS,  is 
not  consistent  with  current  legislative  requirements  and  federal  guidance 
and  has  historically  resulted  in  federal  agencies’  investing  huge  sums  of 
money  in  systems  before  they  realize  that  the  systems  do  not  provide 
commensurate  benefits.  According  to  SPS  program  and  DOD’s  CIO 
officials,  incremental  investment  management  principles  were  not 
employed  because  DOD  policy  did  not  require  them,  and  the  decisions 
regarding  investment  in  SPS  were  made  at  levels  above  the  DOD  CIO’s 
office  without  regard  to  either  the  results  of  the  economic  analyses  or 
progress  against  commitments. 

In  light  of  the  serious  uncertainties  surrounding  further  investment  in  SPS, 
we  are  recommending  that  the  Secretary  of  Defense  make  investment  in 
future  releases  or  major  enhancements  of  existing  releases  conditional 
upon  first  demonstrating  that  existing  releases  are  producing  benefits  that 
exceed  costs  and  that  future  investment  decisions,  including  those 
regarding  operations  and  maintenance  beyond  fiscal  year  2001,  be  based 
on  complete  and  reliable  economic  justifications. 

In  commenting  on  a  draft  of  this  report,  DOD  either  agreed  or  partially 
agreed  with  our  recommendations,  taking  exception  with  those  elements 
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of  our  recommendations  that  the  department  said  would  delay 
development  and  deployment  of  SPS,  specifically,  acquiring  and  using  the 
information  we  believe  is  needed  to  make  informed  investment  decisions. 
To  support  its  position,  however,  DOD  offers  no  new  facts  or  analyses. 
Instead,  DOD  either  cites  information  already  in  our  report  or  claims  that 
the  demands  of  incremental  investment  management  are  “inefficient, 
costly,  and  overly  intrusive”  and  will  cause  “unwarranted  delays  and 
disruption  to  the  program”  for  no  other  reason  than  “to  satisfy  economists 
and  accountants.”  According  to  DOD,  its  latest  SPS  economic  analysis  and 
its  existing  efforts  to  measure  progress  against  selected  program 
commitments  provide  sufficient  bases  for  continuing  to  invest  hundreds  of 
millions  of  dollars  in  SPS. 

We  disagree  with  these  comments.  As  we  describe  in  the  report, 
incremental  investment  management  practices  are  not  only  a  best 
practice,  but  are  also  required  by  the  Clinger-Cohen  Act  of  1996  and 
specified  in  OMB  guidance  and  recently  revised  DOD  acquisition  policy. 
Therefore,  DOD’s  comments  regarding  incremental  investment  in  SPS  are 
at  odds  with  contemporary  practices  and  operative  federal  requirements 
and  guidance.  In  addition,  DOD’s  latest  economic  analysis  is  not  credible 
for  a  number  of  reasons  discussed  in  our  report,  such  as  the  retum-on- 
investment  calculation  highlighted  in  the  analysis  summary  not  including 
relevant  costs  and  program  changes  affecting  system  benefits.  Moreover, 
DOD’s  reported  progress  to  date  against  program  commitments  shows 
that  DOD  is  not  meeting  some  commitments  and  it  does  not  know  if  it  is 
meeting  other  commitments.  For  example,  DOD  does  not  yet  know 
whether  SPS  is  meeting  benefit  expectations.  Our  recommendations  are 
aimed  at  ensuring  that  DOD  obtains  the  information  it  needs  to  make 
informed  SPS  investment  decisions. 

DOD  provided  other  clarifying  comments  that  have  been  incorporated  as 
appropriate  throughout  this  report.  The  written  comments,  along  with  our 
responses,  are  reproduced  in  appendix  II. 


Background 


DOD’s  procurement  process  spans  numerous  Defense  agencies  and 
military  services.  This  process  provides  for  acquiring  supplies  and  services 
from  nonfederal  sources  and,  when  necessary,  administering  the  related 
contractual  instruments.  It  also  provides  for  administering  grants, 
cooperative  agreements,  and  other  transactions  executed  by  contracting 
offices.  The  procurement  process  begins  with  the  receipt  of  a  requirement 
and  ends  at  the  contract  closeout.  (See  fig.  1  for  a  simplified  diagram  of 
the  procurement  process,  the  interaction  of  this  process  with  the  logistics 
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and  financial  management  processes,  and  those  functions  within  the 
procurement  process  that  SPS  is  to  support.) 


Figure  1:  DOD’s  Procurement  Process  and  Functions  to  be  Supported  by  SPS 
(Darker  Shading)  Linked  to  Logistics  and  Financial  Management  Processes  (Lighter 
Shading) 
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Source:  DOD. 


SPS:  A  Brief  Description  In  November  1994,  DOD’s  Director  of  Defense  Procurement  (DDP) 

initiated  the  SPS  program  to  acquire  and  deploy  a  single  automated  system 
to  perform  all  contract  management-related  functions  within  DOD’s 
procurement  process  for  all  DOD  organizations  and  activities.  From  1994 
to  1996,  DOD  defined  SPS  requirements  and  solicited  commercially 
available  vendor  products  for  satisfying  these  requirements.  DOD 
subsequently  awarded  a  contract  to  American  Management  Systems 
(AMS),  Incorporated,  in  April  1997,  to  (1)  use  its  commercially  available 
contract  management  system  as  the  foundation  for  SPS,  (2)  modify  this 
commercial  product  as  necessary  to  meet  the  requirements,  and  (3) 
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perform  related  services.5  DOD  also  directed  the  contractor  to  deliver  SPS 
functionality  in  four  incremental  releases.  The  department  later  increased 
the  number  of  releases  across  which  this  functionality  would  be  delivered 
to  seven;  reduced  the  size  of  the  increments;  and  allowed  certain,  more 
critical  functionality  to  be  delivered  sooner. 

Over  the  last  4  years,  DOD  and  AMS  have  deployed  four  releases  to  773 
locations  in  support  of  21,900  users.  The  fifth  release  was  delivered  in 
February  2001  for  acceptance  testing;  however,  due  to  software 
deficiencies,  this  release  was  sent  back  to  the  vendor  for  rework  and  has 
not  been  deployed.  AMS  is  expected  to  provide  a  second  version  of  this 
release  to  DOD  in  July  2001  for  additional  testing.  If  accepted,  the  fifth 
release  is  to  be  deployed  to  about  4,500  users  beginning  in  fiscal  year  2002. 
DOD  has  not  yet  contracted  for  the  sixth  and  seventh  releases.  (See  table  2 
for  the  status  of  the  various  software  releases,  and  table  3  for  the  summary 
of  SPS  functionality  by  increment.) 


According  to  a  program  official,  DOD  is  not  acquiring  the  source  code  for  SPS  and,  unless 
an  expanded  license  is  obtained,  is  required  to  obtain  sole-source  support  over  the  life  of 
this  system  from  AMS. 
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Table  1 :  Summary  of  SPS  Status  by  Software  Release/Version 


Software 


Increment 

release/version 

Original  delivery  date 

Revised  delivery  date 

Actual  delivery  date 

Schedule  slippage 

1 

3.1 

September  1996 

— 

September  1996 

None 

2 

3.5 

December  1996 

— 

December  1996 

None 

3 

4.0 

February  1998 

— 

March  1998 

1  month 

4" 

4.1b 

N/A 

September  1998 

July  2000 

22  months 

5a 

4.2bc 

N/A 

February  2001 

— 

5  months 

6 

5.0 

February  1999 

Not  under  contract" 

N/A 

N/A 

T 

5.1b 

N/A 

Not  under  contract" 

N/A 

N/A 

Note:  N/A,  or  not  applicable,  because  it  is  not  one  of  the  original  four  increments. 

"Increment  added  to  original  four  increments. 

"Versions  4.1  and  4.2  provide  functionality  initially  scheduled  to  be  delivered  in  version  5.0.  Program 
officials  decided  that  this  functionality  was  needed  sooner  than  was  previously  scheduled  for 
deployment  to  new  user  communities.  Version  5.1  will  also  contain  functionality  previously  planned  to 
be  incorporated  in  version  5.0. 

"Implementation  of  version  4.2  would  enable  base-level  contracting,  often  referred  to  as  vendor 
payment  functionality,  which  accounts  for  approximately  95  percent  of  DOD  contracting  activities  and 
56  percent  of  total  contracting  costs. 

"As  of  June  27,  2001 ,  DOD  had  not  entered  into  a  contractual  arrangement  with  the  vendor  for 
delivery  of  this  release. 

Source:  DOD. 


Table  2:  Summary  of  SPS  Functionality  by  Increment 

Increment 

Functionality 

1 

Provide  base-level  contracting  capabilities  enabling  users  to  prepare 
simple  contracts,  which  are  generally  fixed  price,  1-year  contracts  that 
will  not  be  modified. 

2-5 

Provide  enhanced  base-level  contracting  functionality,  such  as 
reporting  and  contract  administration  capabilities,  automatic  edits, 
security  features,  and  electronic  interfaces  for  legacy  systems  being 
replaced. 

6 

Provide  more  complex  contracting  capabilities,  enabling  the  purchase 
of  weapon  systems.  These  contracts  are  more  complicated,  consisting 
of  numerous  provisions  and  contract  line  item  numbers,  and  usually 
undergo  extensive  modifications. 

7 

Provide  inventory  control  point  functionality  enabling  workload 
management  to  better  manage  inventories. 

Source:  DOD. 


As  planned,  SPS  is  to  be  used  to  prepare  contracts  and  contract-related 
documents  and  to  support  contracting  staff  in  monitoring  and 
administering  them.  SPS  also  is  intended  to  standardize  procurement 
business  practices  and  data  elements  throughout  DOD  and  to  provide 
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timely,  accurate,  and  integrated  contract  information.  Using  SPS,  the  goal 
is  that  required  contract  and  contract  payment  data  will  be  entered  once — 
at  the  source  of  the  data’s  creation — and  be  stored  in  a  single  database.  As 
depicted  in  figure  1,  SPS  is  to  electronically  interface  with  DOD’s  logistics 
community,  which  is  the  source  of  goods  and  services  requests,  and  with 
the  Defense  Finance  and  Accounting  Service  (DFAS),  which  is  responsible 
for  contract  payments.6 

DDP  is  organizationally  within  the  Office  of  the  Under  Secretary  of 

Defense  for  Acquisition,  Technology  and  Logistics.  However,  as  shown  in 
table  3,  the  management  responsibility  for  SPS  is  shared  among  several 
organizations. 

Table  3:  Summary  of  SPS  Management  Responsibilities  and  Functions 

Entity 

Responsibility /Function 

Director  of  Defense  Procurement 

Has  overall  responsibility  for  SPS;  established  an  SPS  Program  Management  Office. 

SPS  Program  Management  Office3 

Although  delegated  responsibility  for  SPS,  does  not  manage  all  aspects  of  the  program, 
not  even  the  total  budget.  (The  Program  Management  Office  has  its  own  budget.) 

Manages  and  deploys  SPS,  as  well  as  monitors  contract  performance.  Performs 
acceptance  testing  and  training  of  the  product  delivered  by  the  contractor  and  addresses 
issues  before  the  software  is  provided  to  the  various  DOD  organizations  for 
implementation. 

Various  DOD  components/ 
organizations11 

Provide  component-level  configuration  control,  data  migration,  standard  interface 
requirements,  training,  site  migration,  and  transition  from  legacy  systems  to  SPS. 

As  a  result,  each  component  maintains  its  own  separate  budget  for  acquisition  of  SPS 
infrastructure,  integration  of  SPS  software  and  infrastructure,  site  acceptance  and 
operational  testing,  etc. 

Assistant  Secretary  of  Defense  for 
Command,  Control,  Communications, 
and  Intelligence/Chief  Information 
Officer 

(ASD/C3I/CIO) 

The  milestone  decision  authority  for  SPS,  which  is  categorized  as  a  major  automated 
information  system  initiative. 

As  the  milestone  decision  authority,  the  CIO  approves  the  program  to  proceed  through  its 
acquisition  cycle  on  the  basis  of  a  review  of  key  documents  (e.g.,  an  independently 
evaluated  life-cycle  cost-and-benefits  estimate,  a  component  cost  analysis,  and  an 
acquisition  strategy  and  program  baseline). 

“The  SPS  program  office  is  within  the  Defense  Contract  Management  Agency. 

"For  example,  the  Departments  of  the  Army,  Navy,  and  Air  Force,  and  the  Defense  Logistics  Agency. 


Source:  DOD. 


6DFAS  is  to  use  the  information  captured  during  the  contract  administration  and  delivery 
and  contract  functions  as  the  basis  on  which  to  make  payments  for  goods  and/or  services 
received. 
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Previous  Reviews  of  SPS 
Have  Identified 
Management  Concerns 


Since  1996,  DOD’s  Office  of  the  Inspector  General  (OIG)  has  issued  three 
reports  critical  of  SPS.  In  September  1996,  the  OIG  reported  that  the  needs 
of  SPS  users  might  not  be  met  and  that  actual  costs  could  exceed 
proposed  costs  because,  among  other  things,  the  functional  requirements 
were  very  broad,  existing  commercial  software  required  substantial 
modification,  and  adequate  development  and  operational  test  strategies 
had  not  been  developed.7 

The  OIG  later  reported  in  May  1999  that  SPS  lacked  critical  functionality 
and  concluded  that  the  system  may  not  meet  mission  needs  with  regard  to 
standardizing  procurement  policy,  processes,  and  procedures.8  The  report 
also  noted  that  users  were  receiving  inadequate  system  training,  guidance, 
and  support,  thereby  forcing  users  to  develop  inefficient  system 
workarounds.  Finally,  the  report  raised  concerns  about  the  cost- 
effectiveness  of  DOD’s  contractual  reliance  on  a  single  vendor  to  provide 
system  support  over  the  life  of  SPS,  adding  that  an  expanded  license  was 
needed  to  give  DOD  the  ability  to  competitively  compete  support  services. 

In  March  2001,  the  OIG  reported  that  lack  of  system  functionality  was  still 
a  serious  program  concern,  productivity  had  not  increased  with  the 
implementation  of  version  4.1,  and  users  were  generally  dissatisfied  with 
SPS.9 

SPS  program  officials  generally  concurred  with  the  OIG’s  findings  and 
agreed  to 

•  issue  guidance  on  the  acquisition  of  commercial  software  for  major 
automated  information  systems, 

•  support  development  of  accurate  life-cycle  cost  estimates  for  SPS, 

•  clarify  responsibilities  for  the  program  office,  the  contractor,  and  the 
DOD  component  organizations, 

•  evaluate  the  cost  and  benefits  of  obtaining  additional  license  rights  and 
renegotiating  the  contract, 


7 Allegations  to  the  Defense  Hotline  Concerning  the  Standard  Procurement  System,  Office 
of  the  Inspector  General,  Department  of  Defense  (Report  No.  96-219,  September  5,  1996). 

8 Initial  Implementation  of  the  Standard  Procurement  System,  Office  of  the  Inspector 
General,  Department  of  Defense  (Report  No.  99-166,  May  26,  1999). 

8 'Standard  Procurement  System  Use  and  User  Satisfaction,  Office  of  the  Inspector 
General,  Department  of  Defense  (Report  No.  D-2001-075,  March  13,  2001). 
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•  require  the  program  office  to  be  aware  of  additional  support  contracts, 
and  suggest  that  the  component  organizations  provide  funds  to  the 
program  office  to  better  integrate  user  needs, 

•  better  coordinate  training  needs  among  the  DOD  component 
organizations,  and 

•  require  that  before  any  future  deployments  of  SPS,  the  DOD 
component  organizations  determine  that  the  version  meets  their 
functional  requirements  and  to  identify  the  number  of  licenses 
required. 

Also,  in  response  to  the  OIG’s  March  2001  report,  the  SPS  program  office 
initiated  its  own  study  in  June  2000  to  assess  the  extent  to  which  benefits 
will  be  realized  as  a  result  of  its  implementation  of  version  4.1  of  SPS.  The 
program  office  plans  to  publish  the  study  results  by  October  2001. 


DOD  Either  Is  Not 
Meeting  or  Does  Not 
Know  If  It  Is  Meeting 
SPS  Commitments 


Federal  information  technology  (IT)  investment  management 
requirements  and  guidance  recognize  the  need  to  measure  investment 
programs’  progress  against  commitments.  In  the  case  of  SPS,  DOD  is  not 
meeting  key  commitments  and  is  not  measuring  whether  it  is  meeting 
other  commitments.  According  to  the  program  manager,  the  program 
office  is  not  responsible  for  ensuring  that  all  program  commitments  are 
being  met.  Rather,  the  program  office’s  sole  task  is  to  acquire  and  deploy 
an  SPS  system  solution  that  meets  defined  functional  requirements.  Given 
that  SPS  is  a  major  Defense  acquisition,10  the  DOD  CIO  is  the 
decisionmaking  authority  for  SPS.  However,  according  to  officials  in  the 
CIO’s  office,  SPS  has  continued  to  be  approved  and  funded  regardless  of 
progress  against  expectations  on  the  basis  of  decisions  made  by 
individuals  organizationally  above  the  CIO’s  office.  Without  measuring  and 
reporting  progress  against  program  commitments  and  taking  the 
appropriate  actions  to  address  significant  deviations,  DOD  runs  the 
serious  risk  of  investing  billions  of  dollars  in  a  system  that  will  not 
produce  commensurate  value. 


10DOD  Regulation  5000.2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  and  Major  Automated  Information  System  Acquisition  Programs,  specifies 
mandatory  policies  and  procedures  for  major  acquisitions.  The  policy  also  specifies  that 
the  DOD  CIO  is  the  milestone  decision  authority,  responsible  for  program  approval,  for  all 
major  automated  information  systems,  such  as  SPS. 
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Measuring  Progress 
Against  Program 
Commitments  Is  a  Key  to 
Effective  Investment 
Management 


The  Clinger-Cohen  Act  of  1996  and  Office  of  Management  and  Budget 
(OMB)  guidance11  emphasize  the  need  to  have  investment  management 
processes  and  information  to  help  ensure  that  IT  projects  are  being 
implemented  at  acceptable  costs  and  within  reasonable  and  expected  time 
frames  and  that  they  are  contributing  to  tangible,  observable 
improvements  in  mission  performance  (i.e.,  that  projects  are  meeting  the 
cost,  schedule,  and  performance  commitments  upon  which  their  approval 
was  justified).  For  programs  such  as  SPS,  DOD  requires  this  cost, 
schedule,  and  performance  information  to  be  reported  quarterly  to  ensure 
that  programs  do  not  deviate  significantly  from  expectations.12  In  effect, 
these  requirements  and  guidance  recognize  that  one  cannot  manage  what 
one  cannot  measure. 


DOD  Has  Not  Met  Key  SPS  DOD  has  not  met  key  SPS  commitments  concerning  the  timing  of  product 
Program  Commitments  delivery,  user  satisfaction  with  system  performance,  and  the  use  of  a 

commercial  system  solution,  as  discussed  below: 

•  DOD  committed  to  SPS’  being  fully  operational  at  all  sites  by  March  31, 
2000;  however,  this  date  has  slipped  by  3- '/a  years  and  is  likely  to  slip 
further. 

Currently,  DOD  has  established  a  September  30,  2003,  milestone  for 
making  SPS  fully  operational,  and  the  program  manager  attributed  this 
delay  to  (1)  problems  encountered  in  modifying  and  testing  the 
contractor’s  commercial  product  to  meet  DOD’s  requirements  and  (2) 
an  increase  in  requirements.  However,  the  SPS  Joint  Requirements 
Board  chairperson  stated  that  no  additional  requirements  have  been 
approved.  Instead,  the  original  requirements  were  clarified  for  the 
contractor  to  better  ensure  that  the  needs  of  the  user  would  be  met. 

However,  satisfying  even  this  revised  commitment  will  be  problematic 
for  several  reasons.  First,  the  2003  milestone  does  not  recognize  DOD 
components’  testing  activities  that  need  to  occur  before  the  system 


^Clinger-Cohen  Act  of  1996,  Public  Law  104-106  and  Office  of  Management  and  Budget 
(OMB)  Circular  A-130,  Management  of  Federal  Information  Resources  (November  30, 
2000). 

12DOD  Interim  Regulation  5000.2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  and  Major  Automated  Information  System  Acquisition  Programs  (January  4, 
2001). 
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could  be  fully  operational.  For  example,  Department  of  the  Air  Force 
officials  told  us  that  they  are  typically  6  to  12  months  behind  the 
program  office’s  deployment  milestones  because  of  additional  testing 
that  the  Air  Force  performs  before  it  implements  the  software  releases. 
Second,  the  2003  milestone  has  not  been  updated  to  reflect  the  impact 
of  events.  For  example,  version  4.1,  the  latest  deployed  release,  was 
recently  changed  from  a  single  release  to  five  subreleases  to  correct 
software  problems  discovered  during  operation  of  version  4.1;  and 
version  4.2  recently  failed  acceptance  testing,  and  the  vendor  is  still 
attempting  to  correct  identified  defects.  Third,  the  official  responsible 
for  SPS  independent  operational  test  and  evaluation,  as  well  as  the 
official  in  DOD’s  Office  of  Program  Analysis  and  Evaluation  who  is 
responsible  for  reviewing  the  SPS  economic  analyses,  told  us  that  this 
milestone  is  likely  to  slip  further.  The  reasons  that  these  officials  cited 
included  incomplete  system  functionality,  increased  system 
complexity,  and  inadequate  training. 

•  DOD  committed  to  SPS’  satisfying  the  needs  of  its  contracting 
community  and  meeting  specified  system  requirements,  ultimately 
increasing  contracting  efficiency  and  effectiveness.  However, 
according  to  a  recent  DOD  OIG  report,  approximately  60  percent  of  the 
user  population  surveyed  was  not  satisfied  with  the  system’s 
functionality  and  performance,  resulting  in  the  continued  use  of  legacy 
systems  and/or  manual  processes  in  lieu  of  SPS.13 

Similarly,  another  DOD  report  describes  SPS  as  unstable  because  the 
system  frequently  goes  down,  meaning  that  it  is  unexpectedly 
unavailable  to  users  who  are  logged  on  and  using  the  system,  which,  in 
turn,  causes  users  to  lose  information.14  The  report  also  notes  that 
users  complained  that  previously  identified  problems  were  not  being 
resolved  in  later  software  releases,  and  that  requested  changes  or 
enhancements  were  not  being  made.  According  to  the  program 
manager,  at  any  one  time,  there  was  a  backlog  of  100  to  200  problems 
that  needed  to  be  addressed  in  order  for  SPS  to  meet  specified 
requirements.  In  light  of  these  challenges  in  meeting  requirements  and 
satisfying  user  needs,  the  official  responsible  for  independent 


13DOD  Office  of  the  Inspector  General,  Report  No.  D-2001-075,  March  13,  2001. 

14 Standard  Procurement  System,  Increment  3  Operational  Assessment  2000  Report, 
Defense  Information  Systems  Agency,  Joint  Interoperability  Test  Command  (September 
2000). 
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operational  test  and  evaluation  of  SPS  said  that  DOD  should  not  invest 
in  additional  releases  beyond  version  4.2. 16 

•  In  delivering  SPS,  DOD  was  to  use  a  commercially  available  software 
product.  However,  the  contractor  has  modified  the  commercial  product 
extensively  in  an  attempt  to  satisfy  DOD’s  needs;  thus,  SPS  is  now  a 
DOD-unique  system  solution. 

According  to  the  program  manager,  DOD  knew  when  it  selected  the 
commercial  product  that  the  product  provided  only  45  percent  of  the 
functionality  that  DOD  needed,  and  that  extensive  new  software 
development  and  existing  software  modification  were  necessary. 
Nevertheless,  the  product  was  chosen  because  no  commercial  product 
was  available  that  met  DOD’s  requirements,  and,  of  the  products 
available,  DOD  believed  that  AMS’  product  and  company  would 
provide  the  best  value. 

In  accordance  with  industry  best  practices,  software  modifications  to  a 
commercial  product  should  not  exceed  10  to  15  percent.  Beyond  this 
degree  of  software  change,  experts  generally  consider  development  or 
acquisition  of  a  custom  system  solution  more  cost-effective.  Further, 
DOD  guidance  states  that  custom  modifications  to  a  commercial  item, 
even  if  made  and  implemented  by  the  commercial  item’s  vendor,  result 
in  custom  system  solutions.16  This  guidance  emphasizes  the  use  of 
commercial  items  to  reduce  life-cycle  costs  and  increase  product 
reliability  and  availability.  Since  SPS  is  not  a  commercial  product,  DOD 
will  not  be  able  to  take  advantage  of  the  reduced  cost  and  risk 
associated  with  using  proven  technology  that  is  used  by  a  wide 
customer  base. 


^Implementation  of  version  4.2  would  enable  base-level  contracting,  often  referred  to  as 
vendor  payment  functionality,  which  accounts  for  approximately  95  percent  of  DOD 
contracting  activities  and  56  percent  of  total  contracting  costs. 

16DOD  white  paper  entitled  Commercial  Item  Acquisition:  Considerations  and  Lessons 
Learned  (June  26,  2000). 
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DOD  Does  Not  Know  What 
Progress  Is  Being  Made 
Against  Other  SPS 
Commitments 


When  it  began  the  program,  DOD  promised  that  SPS  would  produce  such 
benefits  as  (1)  replacing  76  legacy  systems  and  manual  processes  with  a 
single  system  and  thereby  reducing  procurement  system  operations  and 
maintenance  costs  by  an  unspecified  amount,  (2)  standardizing  policies, 
processes,  and  procedures  across  the  Department,  and  (3)  reducing 
problem  disbursements.17  However,  DOD  does  not  know  the  extent  to 
which  SPS  is  meeting  each  of  these  expectations,  even  though  versions 
have  been  deployed  to  about  773  user  locations. 


First,  although  DOD  reports  that  it  has  retired  two  major  legacy  systems, 
neither  the  program  office  nor  the  DOD  CIO  office  could  provide  us  with 
information  on  what,  if  any,  savings  have  been  realized  by  doing  so. 
Additionally,  program  officials  told  us  that  the  number  of  legacy  systems 
and  manual  processes  that  SPS  is  to  replace  is  now  significantly  less  than 
the  76  originally  used  to  justify  the  program.  In  response  to  our  inquiry,  the 
SPS  program  manager  recently  surveyed  the  DOD  component 
organizations  to  determine  the  number  of  legacy  systems.  According  to  the 
results  of  the  survey,  there  were  55  legacy  procurement  systems.  See  table 
4  for  the  status  of  these  systems  as  of  June  2001.  According  to  the  SPS 
program  manager,  45  of  the  55  systems  remain,  and  10  to  12  of  these 
systems  are  to  be  replaced  by  SPS.  However,  another  program  official 
noted  that  SPS  was  always  intended  to  replace  only  14  major  legacy 
systems.  In  either  case,  the  latest  economic  analysis  has  not  been  updated 
to  reflect  this  change  in  the  number  of  systems  to  be  replaced,  and  the 
associated  cost  savings  are  not  known. 


Table  4:  Status  of  Legacy  Contracting  Systems  as  of  June  2001 

Status 

Number  of 
systems 

Retired  as  a  result  of  SPS  implementation 

2 

Retired  for  reasons  other  than  SPS  implementation 

4 

Will  not  be  retired 

2 

Retirement  dependent  on  implementation  of  a  system  other  than  SPS 

2 

Remaining  legacy  contracting  systems 

45 

Total 

55 

Source:  DOD. 

^Problem  disbursements  result  when  payments  cannot  be  matched  to  contractual 
obligations. 
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Second,  the  standardization  of  policies,  processes,  and  procedures  benefit 
is  not  materializing  because  each  military  service  is  either  in  the  process  of 
developing,  or  has  plans  to  develop,  its  own  unique  policies,  processes, 
and  procedures. 

Third,  program  officials  were  unable  to  provide  evidence  that 
implementing  SPS  has  reduced  problem  disbursements  or  achieved  the 
benefits  outlined  in  the  economic  analysis.  In  fact,  the  latest  economic 
analysis  no  longer  even  cites  reducing  problem  disbursements  as  a  benefit 
because  the  DOD  components’  position  was  that  SPS  would  not 
completely  address  this  problem.  According  to  the  program  manager  and 
CIO  officials,  there  is  no  DOD  policy  that  requires  them  to  assess  whether 
the  expected  benefits  are  in  fact  being  realized. 

When  the  SPS  program  began,  DOD  also  committed  to  a  system  life-cycle 
cost  of  about  $3  billion  over  a  10-year  period.  However,  total  actual 
program  costs  are  not  being  accumulated  and  monitored  against 
estimates,  which  in  2000  were  revised  to  about  $3.7  billion  (a  28-percent 
increase).  Thus,  DOD  does  not  know  what  has  been  spent  on  the  program 
by  all  DOD  component  organizations.  To  date,  the  only  actual  program 
costs  being  collected  and  reported  are  those  incurred  by  the  SPS  program 
office,  which  DOD  reports  to  be  about  $322  million  through  September  30, 
2000. 

To  determine  the  total  cost  of  the  SPS  program  through  September  30, 
2000,  we  requested  cost  information  from  18  Defense  agencies  and  the 
four  military  services.18  These  DOD  components  reported  that  they  have 
collectively  spent  approximately  $125  million  through  September  30,  2000. 
However,  these  reported  costs  are  not  complete  because  (1)  4  of  the  22 


18The  SPS  program  manager  provided  the  list  of  Defense  agencies,  which  we  confirmed 
with  DOD’s  Office  of  the  Inspector  General.  The  agencies  were  the  Defense  Logistics 
Agency,  Defense  Contract  Management  Agency,  Joint  National  Test  Facility,  Defense 
Commissary  Agency,  Real  Estate  and  Facilities  Contracting  Office,  Special  Operations 
Command,  Uniformed  Services  University  of  the  Health  Sciences,  Defense  Threat 
Reduction  Agency,  American  Forces  Information  Services/Television-Audio  Support 
Activity,  Defense  Information  Systems  Agency/Defense  Information  Technology 
Contracting  Organization,  Defense  Microelectronics  Activity,  National  Imagery  and 
Mapping  Agency,  Defense  Intelligence  Agency,  Pentagon  Renovation  Program  Office,  DOD 
Education  Agency,  Ballistic  Missile  Defense  Organization,  Defense  Finance  and  Accoimting 
System,  and  Defense  Advanced  Research  Projects  Agency. 


Page  15 


GAO-Ol-682  DOD  Systems  Modernization 


DOD  components  did  not  respond,19  (2)  components  reported  that  SPS 
costs  were  being  captured  with  other  programs  and  could  not  be  allocated 
accurately,  and  (3)  all  SPS  costs,  such  as  employee  salaries  and  system 
infrastructure  costs,20  were  not  included.  According  to  program  officials, 
no  single  DOD  organization  is  responsible  for  accumulating  the  full  DOD 
cost  of  SPS. 

Without  knowing  the  extent  to  which  SPS  is  meeting  cost-and-benefit 
expectations,  DOD  is  not  in  a  position  to  make  informed,  and  thus 
justified,  decisions  on  whether  and  how  to  proceed  further  on  the 
program.  Such  a  situation  introduces  a  serious  risk  of  investing  in  a  system 
that  will  not  produce  a  positive  net  present  value  (i.e.,  estimated  benefits 
to  be  realized  would  exceed  estimated  program  costs). 


DOD  Has  Not 
Economically 
Justified  Investing  in 
SPS 


Federal  IT  investment  management  requirements  and  guidance,  as  well  as 
DOD  policy,  recognize  the  need  to  economically  justify  IT  projects  before 
investing  in  them  and  to  justify  them  in  an  incremental  manner  in  an  effort 
to  spread  the  risk  of  doing  many  things  over  many  years  on  large  projects 
across  smaller,  more  manageable  subprojects.  However,  the  department 
has  not  economically  justified  investing  in  SPS  because  its  own  analysis 
shows  that  expected  life-cycle  benefits  are  less  than  estimated  life-cycle 
costs.  Moreover,  DOD  is  not  approaching  its  investment  in  SPS  on  an 
incremental  basis.  Nevertheless,  DOD  continues  to  invest  hundreds  of 
millions  of  dollars  in  SPS  each  year,  running  the  serious  risk  of  spending 
large  sums  of  money  on  a  system  that  does  not  produce  commensurate 
value.  According  to  program  and  CIO  officials,  DOD  continues  to  invest 
these  funds  because  individuals  above  the  CIO’s  office  decided  that  SPS 
was  a  departmental  priority. 


19The  Special  Operations  Command,  the  Defense  Threat  Reduction  Agency,  the  Defense 
Intelligence  Agency,  and  the  Ballistic  Missile  Defense  Organization  did  not  provide  cost 
information. 

20Costs  for  various  items,  such  as  desktop  computers,  servers,  and  networks,  were 
excluded  because  they  were  being  used  to  support  other  applications,  as  well  as  SPS. 
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Reliable  Economic 
Analysis  and  Incremental 
Investment  Are  Tenets  of 
Effective  IT  Investment 
Management 


The  Clinger-Cohen  Act  of  1996  and  OMB  guidance  provide  an  effective 
framework  for  IT  investment  management.21  Together,  they  set 
requirements  for  (1)  economically  justifying  proposed  projects  on  the 
basis  of  reliable  analyses  of  expected  life-cycle  costs,  benefits,  and  risks, 
(2)  using  these  analyses  throughout  a  project’s  life  cycle  as  the  basis  for 
investment  selection,  control,  and  evaluation  decisionmaking,  and  (3) 
doing  so  for  large  projects  (to  the  maximum  extent  practical)  by  dividing 
them  into  a  series  of  smaller,  incremental  subprojects  or  releases.  By 
doing  so,  the  tremendous  risk  associated  with  investing  large  sums  of 
money  over  many  years  in  anticipation  of  delivering  capabilities  and 
expected  business  value  far  into  the  future  can  be  spread  across  project 
parts  that  are  smaller,  of  a  shorter  duration,  and  capable  of  being  more 
reliably  justified  and  more  effectively  measured  against  cost,  schedule, 
capability,  and  benefit  expectations. 


DOD  policy  also  reflects  these  investment  principles  by  requiring  that 
investments  be  justified  by  an  economic  analysis  and,  more  recently,  that 
investment  decisions  for  major  programs,  like  SPS,  be  made  incrementally 
by  ensuring  that  each  incremental  part  of  the  program  delivers  measurable 
benefit,  independent  of  future  increments.22  According  to  the  policy,  the 
economic  analysis  is  to  reflect  both  life-cycle  cost  and  benefits  estimates, 
including  a  return-on-investment  calculation,  to  demonstrate  that  a 
proposal  to  invest  in  a  new  system  is  economically  justified  before  that 
investment  is  made. 


SPS  Economic  Analyses  DOD  has  developed  three  economic  analyses  for  SPS — one  in  1995  and 
Do  Not  Justify  Acquiring  two  updates  (one  in  1997  and  another  in  2000).  While  the  initial  analysis 
gpg  reflected  a  positive  net  present  value,  the  two  updates  did  not. 

Specifically,  the  1997  analysis  estimated  life-cycle  costs  and  benefits  to  be 
$2.9  billion  and  $1.8  billion,  respectively,  which  is  a  recovery  of  only  62 
percent  of  costs;  the  2000  analysis  showed  even  greater  costs  ($3.7  billion) 
and  fewer  benefits  ($1.4  billion),  which  is  a  recovery  of  only  37  percent  of 
costs  (see  fig.  2). 


21Clinger-Cohen  Act  of  1996,  Public  Law  104-106,  and  OMB  Circular  A-130  (November  30, 
2000). 

22DOD  Interim  Regulation  5000.2-R  and  DOD  Instruction  5000.2,  Change  1,  Operation  of  the 
Defense  Acquisition  System  (January  4,  2001). 
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Figure  2:  SPS  Life-Cycle  Costs  and  Benefits 


Fiscal  Year  1997 


Fiscal  Year  2000 


Benefit 


Benefit 


Source:  DOD  data. 


Nevertheless,  these  data  were  not  reflected  in  the  return-on-investment 
calculation  in  the  analyses  that  were  used  as  the  basis  for  approving  SPS. 
Instead,  this  return-on-investment  calculation  (1)  included  only  those 
costs  estimated  to  be  incurred  by  the  program  office  and  (2)  excluded  the 
SPS  implementation  and  operation  and  maintenance  costs  of  DOD 
agencies  and  military  services.  According  to  program  officials,  the  latter 
costs  were  excluded  because  either  they  would  have  been  incurred 
anyway  or  the  program  office  did  not  require  them.  For  example,  the 
officials  stated  that  the  DOD  agencies  and  military  services  routinely 
upgrade  their  IT  infrastructures  to  support  existing  systems;  therefore, 
they  assumed  that  the  agencies  and  services  would  have  purchased  new 
infrastructures  even  if  SPS  had  not  been  acquired.  Also,  program  officials 
did  not  believe  that  training  paid  for  by  DOD  agencies  and  militaiy 
services  should  be  included  as  a  cost  element  because  this  is  an  elective 
expense  (i.e.,  the  program  management  office  does  not  require  this 
additional  training).  However,  some  DOD  component  officials  told  us  that 
some  of  their  infrastructure  and  other  costs  were  being  incurred  solely  to 
support  implementation  of  SPS.  Using  DOD’s  estimates,  we  calculated 
SPS’  net  present  value  for  fiscal  years  1997  and  2000  to  be  about  negative 
$174  million  and  negative  $655  million,  respectively. 

DOD’s  Office  of  Program  Analysis  and  Evaluation  is  responsible  for, 
among  other  things,  verifying  and  validating  the  reliability  of  economic 
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analyses  for  major  programs,  such  as  SPS,  and  providing  its  results  to  the 
program  approval  authority,  which  in  this  case  is  the  DOD  CIO.  According 
to  Office  of  Program  Analysis  and  Evaluation  officials,  although  the 
economic  analyses  were  reviewed,  there  are  no  written  results  of  these 
reviews.  These  officials  stated,  however,  that  they  orally  communicated 
concerns  about  the  analyses  to  program  officials  and  to  DOD  CIO  officials 
responsible  for  program  oversight  and  control.  They  also  stated  that  while 
they  could  not  recall  specific  issues  discussed,  they  concluded  that  the 
economic  analyses  provided  a  reasonable  basis  for  decisionmaking. 


Latest  SPS  Economic  To  be  useful  for  informed  investment  decisionmaking,  analyses  of  project 

Analysis  Does  Not  Include  costs,  benefits,  and  risks  must  be  based  on  reliable  estimates.  However, 
Reliable  Cost  Data  most  of  the  cost  estimates  in  the  latest  economic  analysis  are  estimates 

carried  forward  from  the  1997  economic  analysis  (adjusted  for  inflation). 
Only  the  costs  being  funded  and  managed  by  the  SPS  program  office, 
which  are  13  percent  of  the  total  life-cycle  cost  in  the  analysis,  were 
updated  in  2000  to  reflect  more  current  contract  estimates  and  actual 
expenditures/obligations  for  fiscal  years  1995  through  1999.  The  costs  to 
be  funded  and  incurred  by  DOD  agencies  and  the  military  services  were 
not  updated  to  account  for  all  program  changes  or  to  incorporate  better 
information.  In  its  review  of  the  2000  economic  analysis,  the  Naval  Center 
for  Cost  Analysis  also  noted  that  the  DOD  agencies  and  the  military 
services’  cost  information,  which  accounted  for  the  majority  of  the 
program’s  overall  costs,  had  not  been  updated. 

In  fact,  only  two  cost  elements  were  updated  for  the  DOD  component 
organizations  in  the  2000  economic  analysis,  and  the  estimates  for  these 
cost  elements  were  based  on  estimates  derived  for  just  one  service  (the 
Air  Force),  and  then  extrapolated  to  all  other  DOD  components. 

According  to  Departments  of  the  Army,  Navy,  and  Air  Force  component 
representatives,  these  original  estimates  of  costs,  as  well  as  benefits,  were 
highly  questionable  at  best.  However,  this  uncertainty  was  not  reflected  in 
the  economic  analysis  by  any  type  of  sensitivity  analysis  (i.e.,  an  analysis 
to  explicitly  present  the  return-on-investment  implications  associated  with 
using  estimates  whose  inherent  imprecision  could  produce  a  range  of 
outcomes).  Such  sensitivity  analysis  would  disclose  for  decisionmakers 
the  investment  risk  being  assumed  by  relying  on  the  calculations 
presented  in  the  economic  analysis. 

According  to  the  SPS  program  manager,  costs  in  the  2000  economic 
analysis  were  not  updated  because  information  for  the  DOD  components 
was  not  readily  available  for  inclusion.  Additionally,  updating  DOD 
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component  costs  was  not  viewed  as  relevant  because  the  return-on- 
investment  calculation  cited  in  the  latest  economic  analysis  did  not 
include  these  costs,  and  the  updated  analysis  was  done  after  DOD 
leadership  had  decided  to  increase  funding  and  continue  the  program. 
However,  by  not  using  economic  analyses  that  are  based  on  reliable  cost 
estimates,  DOD  is  making  uninformed,  and  thus  potentially  unwise, 
multimillion-dollar  investment  decisions. 


Latest  SPS  Economic 
Analysis  Does  Not  Reflect 
Cost  Increases  and  Benefit 
Delays 


According  to  OMB  guidance,23  analyses  of  investment  costs,  benefits,  and 
risks  should  be  (1)  updated  throughout  a  project’s  life  cycle  to  reflect 
material  changes  in  project  scopes  and  estimates  and  (2)  used  as  a  basis 
for  ongoing  investment  selection  and  control  decisions.  To  do  less,  risks 
continued  investment  in  projects  on  the  basis  of  outdated  and  invalid 
economic  justification. 


The  latest  economic  analysis  (January  2000)  is  outdated  because  it  does 
not  reflect  SPS’  current  status  and  known  risks  associated  with  program 
changes.  For  instance,  this  analysis  is  based  on  a  program  scope  and 
associated  costs  and  benefits  that  anticipated  four  software  releases,  each 
providing  more  advanced  features  and  functions.  However,  according  to 
the  program  manager,  SPS  now  consists  of  seven  releases  over  which 
additional  requirements  are  to  be  delivered.  Estimates  of  the  full  costs, 
benefits,  and  risks  relating  to  these  additional  three  releases  are  not  part 
of  this  latest  economic  analysis. 

Also,  the  2000  economic  analysis  does  not  fully  recognize  actual  and 
expected  delays  in  meeting  SPS’  full  operational  capability  milestone.  That 
is,  the  2000  economic  analysis  assumed  that  this  milestone  would  be 
September  30,  2003.  However,  as  previously  mentioned,  this  milestone 
date  is  unlikely  to  be  met  for  a  variety  of  reasons,  such  as  user 
dissatisfaction  with  current  system  capabilities. 

According  to  the  SPS  program  manager,  the  latest  economic  analysis  has 
not  been  updated  to  reflect  changes  because  the  analysis  is  not  used  for 
managing  the  program  and  because  there  is  no  DOD  requirement  for 
updating  an  economic  analysis  when  changes  to  the  program  occur.  By 
not  ensuring  that  the  program  is  being  proactively  managed  on  the  basis  of 


i3OMB  Circular  A-130  (November  30,  2000). 
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current  information  about  costs,  benefits,  and  risks,  DOD  is  unnecessarily 
assuming  an  excessive  amount  of  investment  risk. 

DOD  Is  Not  Investing 
Incrementally  in  SPS 

As  we  have  previously  reported,  incremental  investment  management 
involves  three  fundamental  components:  (1)  developing/acquiring  a  large 
system  in  a  series  of  smaller  projects  or  system  increments,  (2) 
individually  justifying  investment  in  each  separate  increment  on  the  basis 
of  costs,  benefits,  and  risks,  and  (3)  monitoring  actual  benefits  achieved 
and  costs  incurred  on  completed  increments  and  modifying  subsequent 
increments  or  investments  to  reflect  lessons  learned.24 

While  DOD  is  acquiring  and  implementing  SPS  in  a  series  of  incremental 
releases  (originally  four  and  now  seven),  it  is  not  making  decisions  about 
whether  to  invest  in  each  release  on  the  basis  of  the  release’s  costs, 
benefits,  and  risks,  and  it  is  not  measuring  whether  it  is  meeting  cost-and- 
benefit  expectations  for  each  release  that  is  implemented.  Instead,  DOD  is 
treating  investment  in  SPS  as  one,  monolithic  investment  decision, 
justified  by  a  single,  all-or-nothing  economic  analysis.  Moreover,  DOD  has 
not  measured  whether  the  incremental  software  releases  have  produced 
expected  business  value,  even  though  its  economic  analysis  aligns 
expected  benefits  with  the  then  four  incremental  releases. 

In  June  2000,  the  SPS  program  office  initiated  a  study  in  an  attempt  to 
validate  the  extent  to  which  benefits  would  be  realized  as  a  result  of 

DOD’s  implementation  of  version  4.1  of  the  software.  However,  our  review 
of  the  methodology  and  preliminary  results  revealed  that  the  study  was 
poorly  planned  and  executed  and  that,  while  useful  information  may  be 
compiled,  DOD  would  be  unable  to  use  the  study’s  results  to  validate  the 
accrual  of  benefits. 

As  a  result,  DOD  will  have  spent  hundreds  of  millions  of  dollars  on  the 
entire  system  before  knowing  whether  it  is  producing  value  commensurate 
with  cost.  The  program  manager  told  us  that  knowing  whether  SPS  is 
producing  such  value  is  not  the  program  office’s  objective.  Rather,  its 
objective  is  to  simply  acquire  and  deploy  the  system.  Similarly,  DOD  CIO 
officials  told  us  that  although  the  economic  analysis  promised  a  business 
value  that  would  exceed  costs,  DOD  is  not  validating  that  implemented 

2iCustoms  Service  Modernization:  Serious  Management  and  Technical  Weaknesses  Must 
Be  Coin'ected  (GAO/AIMD-99-41,  February  26,  1999). 
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releases  are  producing  that  value  because  there  is  no  DOD  requirement 
and  no  metrics  defined  for  doing  so.25  By  not  investing  incrementally  in 
SPS,  DOD  runs  the  serious  risk  of  discovering  too  late  (i.e.,  after  it  has 
invested  hundreds  of  millions  of  dollars)  that  SPS  is  not  cost-beneficial. 


Conclusions 


DOD’s  management  of  SPS  is  a  lesson  in  how  not  to  justify,  make,  and 
monitor  the  implementation  of  IT  investment  decisions.  Specifically,  DOD 
has  not  (1)  ensured  that  accountability  and  responsibility  for  measuring 
progress  against  commitments  are  clearly  understood,  performed,  and 
reported,  (2)  demonstrated,  on  the  basis  of  reliable  data  and  credible 
analysis,  that  the  proposed  system  solution  will  produce  economic 
benefits  commensurate  with  costs  before  investing  in  it,  (3)  used  data  on 
progress  against  project  cost,  schedule,  and  performance  commitments 
throughout  a  project’s  life  cycle  to  make  investment  decisions,  and  (4) 
divided  this  large  project  into  a  series  of  incremental  investment  decisions 
to  spread  the  risks  over  smaller,  more  manageable  components. 

Currently,  DOD  is  not  effectively  performing  any  of  these  basic  tenets  of 
effective  investment  management  on  SPS,  and,  as  a  result,  DOD  lacks  the 
basic  information  needed  to  make  informed  decisions  about  how  to 
proceed  with  the  project. 

Nevertheless,  DOD  continues  to  push  forward  in  acquiring  and  deploying 
additional  versions  of  SPS.  Continuing  with  this  approach  to  investment 
management  introduces  considerable  risk.  As  a  result,  beyond  possibly 
operating  and  maintaining  already  implemented  releases  for  the  remainder 
of  fiscal  year  2001  and  meeting  already  executed  contractual 
commitments,  further  investment  in  SPS  has  not  been  justified. 


Recommendations  for 
Executive  Action 


We  recommend  that  the  Secretary  of  Defense  direct  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence,  as  the  designated  approval  authority  for  SPS,  to  clarify 
organizational  accountability  and  responsibility  for  measuring  SPS 
progress  against  commitments  and  to  ensure  that  these  responsibilities  are 
met.  We  further  recommend  that  the  Secretary  direct  the  Assistant 
Secretary  to  make  investment  in  each  new  release,  or  each  enhancement 


25In  January  2001,  DOD  issued  a  change  to  its  major  system  acquisition  policy  requiring 
incremental  investment  management.  Specifically,  the  policy  notes  that  a  program’s 
milestone  decision  authority  must  verify  that  each  increment  meets  part  of  the  mission 
need  and  delivers  a  measurable  benefit,  independent  of  future  increments. 
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to  an  existing  release,  conditional  upon  (1)  validating  that  already 
implemented  releases  of  the  system  are  producing  benefits  that  exceed 
costs  and  (2)  demonstrating  on  the  basis  of  credible  analysis  and  data  that 
(a)  proposed  new  releases  or  enhancements  to  existing  releases  will 
produce  benefits  that  exceed  costs  and  (b)  operation  and  maintenance  of 
already  deployed  releases  of  SPS  will  produce  benefits  that  exceed  costs. 

Also,  we  recommend  that  the  Secretary  direct  the  Director,  Program 
Analysis  and  Evaluation,  to  validate  any  analysis  produced  to  justify 
further  investment  in  SPS  and  to  report  any  validation  results  to  the 
Assistant  Secretary  of  Defense  for  C3I.  We  also  recommend  that  no  further 
decisions  to  invest  in  SPS  be  made  without  these  validation  results. 

Additionally,  we  recommend  that  the  Secretary  direct  the  Assistant 
Secretary  of  Defense  for  C3I  to  take  the  necessary  actions,  in  collaboration 
with  the  SPS  program  manager,  to  immediately  determine  the  current 
state  of  progress  against  program  commitments  addressed  in  this  report 
and  to  ensure  that  such  information  is  used  in  all  future  investment 
decisions  concerning  SPS. 

Last,  we  recommend  that  the  Secretary  direct  the  Assistant  Secretary  of 
Defense  for  C3I  to  report  by  October  31,  2001,  to  the  Secretary  and  to 
DOD’s  relevant  congressional  committees  on  lessons  learned  from  the  SPS 
investment  management  experience,  including  what  actions  will  be  taken 
to  prevent  a  recurrence  of  this  experience  on  other  system  acquisition 
programs. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report  (reprinted  in  appendix  II), 
the  Acting  Deputy  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence,  who  is  also  the  DOD  Deputy  Chief 
Information  Officer  (CIO),  agreed  and  partially  agreed  with  our 
recommendations.  In  particular,  the  Deputy  CIO  agreed  with  our 
recommendation  regarding  the  need  to  clarify  organizational 
accountability  and  responsibility  for  measuring  the  program’s  progress 
and  ensuring  that  these  responsibilities  are  met.  The  Deputy  CIO  also 
agreed  to  document  lessons  learned  and  have  the  Director  of  Program 
Analysis  and  Evaluation  validate  the  results  of  any  ongoing  and  future 
analyses  of  SPS’  return  on  investment. 

However,  the  Deputy  CIO  disagreed  with  our  report’s  overall  finding  that 
continued  investment  in  SPS  has  not  been  justified,  and  disagreed  with 
those  elements  of  our  recommendations  that  could  delay  development  and 
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deployment  of  SPS,  specifically,  acquiring  and  using  the  information  we 
believe  is  needed  to  make  informed  investment  decisions.  To  support  its 
position,  however,  the  Deputy  CIO  offered  no  new  facts  or  analyses. 
Instead,  the  comments  either  cite  information  already  in  our  report  or 
claims  that  the  demands  of  incremental  investment  management  are 
“inefficient,  costly,  and  overly  intrusive”  and  will  cause  “unwarranted 
delays  and  disruption  to  the  program”  for  no  other  reason  than  “to  satisfy 
economists  and  accountants.”  According  to  DOD’s  comments,  the  latest 
SPS  economic  analysis  and  the  existing  efforts  to  measure  progress 
against  selected  program  commitments  provide  sufficient  bases  for 
continuing  to  invest  hundreds  of  millions  of  dollars  in  SPS.  In  particular, 
DOD  stated  that  it  is  making  progress  in  improving  its  ability  to 
standardize  contracting  for  goods  and  services,  adding  that  this 
standardization  progress  is  not  only  saving  operating  costs  by  retiring 
legacy  procurement  systems,  but  is  also  providing  a  standard  environment 
within  DOD  for  the  exchange  of  information  and  a  consistent  look  and  feel 
of  contract  information  to  companies  doing  business  with  the  department. 
In  light  of  these  outcomes,  DOD  commented  that  one  of  its  main  goals 
under  the  program  is  the  timely  fielding  of  SPS  capability. 

We  disagree  with  these  comments.  As  we  describe  in  the  report, 
incremental  investment  management  practices  are  not  only  a  best 
practice,  but  are  also  required  by  the  Clinger-Cohen  Act  of  1996  and 
specified  in  OMB  guidance  and  recently  revised  DOD  acquisition  policy. 
Therefore,  DOD’s  comments  regarding  incremental  investment  in  SPS  are 
at  odds  with  contemporary  practices  and  operative  federal  requirements 
and  guidance. 

Additionally,  the  economic  analysis  that  DOD’s  comments  refer  to  is  not 
reliable  for  a  number  of  reasons  that  are  discussed  in  our  report. 
Specifically,  this  analysis  treats  SPS  as  a  single,  monolithic  system 
investment.  Experience  has  shown  that  such  an  all-or-nothing  economic 
justification  is  too  imprecise  to  use  in  making  informed  decisions  on  large 
investments  that  span  many  years.  This  kind  of  approach  for  justifying 
investment  decisions  has  historically  resulted  in  agencies’  investing  huge 
sums  of  money  in  systems  that  do  not  provide  commensurate  benefits,  and 
thus  has  been  abandoned  by  successful  organizations.  Further,  the  need  to 
avoid  this  pitfall  was  a  major  impetus  for  the  Clinger-Cohen  Act 
investment  management  reforms. 

Also,  as  discussed  in  our  report,  the  analysis  highlights  a  return-on- 
investment  calculation  in  its  summary  that  does  not  include  all  relevant 
costs,  such  as  the  costs  to  be  incurred  by  DOD  components.  Instead,  the 
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summary  uses  only  SPS  program  office  costs  in  this  return-on-investment 
calculation.  Further,  this  return-on-investment  calculation  does  not  reflect 
known  changes  in  the  program’s  scope  and  schedule  that  would  increase 
costs  and  reduce  benefits.  As  our  report  points  out,  it  does  not,  for 
example,  reflect  SPS’  change  from  four  software  releases  to  seven  releases 
nor  does  it  reflect  the  improbability  of  meeting  a  September  30,  2003,  full 
operational  capability  date. 

DOD’s  comments  also  promote  continued  spending  on  SPS  without 
sufficient  awareness  of  progress  against  meaningful  commitments,  such  as 
reliable  data  measuring  and  validating  whether  return-on-investment 
projections  are  being  met.  In  fact,  DOD’s  comments  emphasize 
standardization  and  fast  deployment  as  core  commitments.  However, 
neither  factor  is  an  end  in  and  of  itself.  Unless  SPS  provides  the  capability 
to  perform  procurement  and  contracting  functions  better  and/or  cheaper, 
and  does  so  to  a  degree  that  makes  SPS  a  more  attractive  investment 
relative  to  the  department’s  other  investment  options,  DOD  will  not  have 
adequate  justification  for  investing  further  in  SPS.  As  our  report 
demonstrates,  the  department  presently  does  not  have  the  information  it 
needs  to  know  whether  this  investment  is  justified,  and  the  information 
that  is  available  raises  serious  questions  about  SPS’  acceptance  by  its  user 
community  and  its  business  value.  Nevertheless,  DOD’s  comments 
indicate  its  intention  to  implement  SPS  as  planned.  Our  recommendations 
are  aimed  at  ensuring  that  the  department  obtains  the  information  it  needs 
to  make  informed  SPS  investment  decisions  before  proceeding  with 
additional  acquisitions. 

DOD  provided  other  clarifying  comments  that  have  been  incorporated  as 
appropriate  throughout  this  report.  The  written  comments,  along  with  our 
responses,  are  reproduced  in  appendix  II. 


As  agreed  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
date  of  this  letter.  At  that  time,  we  will  send  copies  to  the  Chairmen  and 
Ranking  Minority  Members  of  the  Senate  Committee  on  Armed  Services; 
Senate  Appropriations  Subcommittee  on  Defense;  House  Armed  Services 
Committee;  House  Appropriations  Subcommittee  on  Defense; 
Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  House  Committee  on  Government  Reform; 
and  Subcommittee  on  National  Security,  Veterans  Affairs,  and 
International  Relations,  House  Committee  on  Government  Reform.  We  are 
also  sending  copies  of  this  report  to  the  Director,  Office  of  Management 
and  Budget;  the  Secretary  of  Defense;  the  Acting  Secretary  of  the  Army; 
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the  Acting  Secretary  of  the  Navy;  the  Acting  Secretary  of  the  Air  Force;  the 
Acting  Assistant  Secretary  of  Defense  Command,  Control, 
Communications,  and  Intelligence/Chief  Information  Officer;  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics;  the 
Principal  Deputy  and  Deputy  Under  Secretary  for  Management  Reform; 
the  Acting  Director  of  Operational  Testing  and  Evaluation;  the  Director  of 
Program  Analysis  and  Evaluation;  the  Director  of  Defense  Procurement; 
the  Director  of  the  Defense  Contract  Management  Agency;  and  the 
Director  of  the  Defense  Logistics  Agency. 

If  you  have  any  questions  on  matters  discussed  in  this  report,  please  call 
me  at  (202)  512-3439  or  Cynthia  Jackson,  Assistant  Director,  at 
(202)  512-5086.  We  can  also  be  reached  by  e-mail  at  hiter@gao.gov  and 
jacksonc@gao.gov,  respectively.  Key  contributors  to  this  assignment  are 
listed  in  appendix  III. 

Sincerely  yours, 


Randolph  C.  Hite 

Director,  Information  Technology  Systems  Issues 


Page  26 


GAO-Ol-682  DOD  Systems  Modernization 


Appendix  I:  Objectives,  Scope,  and 
Methodology 


Our  objectives  were  to  determine  the  progress  that  the  Department  of 
Defense  (DOD)  has  made  against  the  Standard  Procurement  System  (SPS) 
program  commitments  and  whether  DOD  has  economically  justified 
further  investment  in  SPS. 

To  determine  the  progress  made,  we  first  analyzed  relevant  legislative  and 
Office  of  Management  and  Budget  (OMB)  requirements,  associated  federal 
guidance,  and  applicable  DOD  policy  and  guidance  on  investment 
management.  We  then  analyzed  relevant  program  management  documents 
and  interviewed  program  officials  to  identify  estimates  and  expectations 
for  SPS’  cost,  schedule,  and  performance,  including  the  system  capabilities 
to  be  provided  and  benefits  to  be  produced  by  these  capabilities.  Source 
documents  for  this  information  included,  but  were  not  limited  to,  the 
acquisition  strategy  and  program  baseline,  acquisition  decision 
memorandums,  and  the  quarterly  Defense  Acquisition  Executive 
Summary  report. 

We  then  reviewed  program  management  reports  and  briefings,  interviewed 
program  officials,  and  solicited  information  from  the  various  DOD 
component  organizations  participating  in  SPS’  implementation  to 
determine  reported  cost,  schedule,  and  performance  status.  We  compared 
this  information  against  estimates  and  expectations  to  identify  any 
variances.  We  did  not  independently  validate  the  status  information  that 
we  obtained.  In  cases  where  variances  were  found  or  status  information 
was  not  available,  we  questioned  program  management  and  DOD’s  Office 
of  the  Chief  Information  Officer  (CIO)  oversight  officials.  The  DOD 
organizations  that  were  part  of  our  scope  of  contacts  included  the  SPS 
program  office  within  the  Defense  Contract  Management  Agency;  the 
Office  of  the  Director  of  Investments  and  Acquisition  within  the  Office  of 
the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence  (C3I)/Chief  Information  Officer;  the 
Office  of  the  Deputy  Director  (Strategic  and  Space  Programs)  within  the 
Office  of  Program  Analysis  and  Evaluation  under  the  Office  of  the 
Undersecretary  of  Defense  (Comptroller/Chief  Financial  Officer);  the 
Office  of  Strategic  and  C3I  Systems  within  the  Office  of  the  Director  of 
Operational  Test  and  Evaluation;  and  various  offices  within  the  Defense 
agencies  and  military  services  responsible  for  implementing  SPS. 

To  determine  whether  DOD  had  economically  justified  SPS,  we  reviewed 
relevant  legislative  requirements  and  associated  OMB  guidance,  as  well  as 
DOD  policy  and  guidance  on  preparing  and  using  economic  analyses  (cost, 
benefit,  and  risk),  to  measure  progress  against  information  technology  (IT) 
investment  decisions  and  to  do  so  using  an  incremental  or  modular 
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approach.1  We  then  obtained  the  original  economic  analyses  prepared  for 
the  program  and  the  two  subsequent  updates  and  evaluated  them  in  light 
of  relevant  requirements,  policies,  and  guidance  to  identify  strengths  and 
weaknesses.  We  also  reviewed  program  management  documents  and 
interviewed  program  and  oversight  officials  to  understand  how  these 
analyses  were  reviewed  and  used,  and  we  compared  the  results  to  relevant 
requirements  and  guidance.  We  also  calculated  the  program’s  net  present 
value  using  the  1997  and  2000  economic  analyses.2  In  addition,  we 
interviewed  officials  from  the  SPS  program  office,  DOD  CIO’s  office,  and 
DOD’s  Program  Analysis  and  Evaluation  Office  to  discuss  our  results  and 
seek  clarifying  information.  We  reviewed  the  methodology  and  preliminary 
results  for  the  productivity  study  being  conducted  by  DOD  to  substantiate 
the  benefits  to  be  realized  by  implementing  SPS.  We  also  interviewed 
officials  from  the  SPS  program  office,  Vector  Research  Incorporated,  and 
Logistics  Management  Institute  to  discuss  the  methodology  (e.g.,  survey 
execution,  sampling,  and  analysis  plans)  and  our  conclusions  on  the  study. 

We  conducted  our  work  at  DOD  headquarters  offices  in  Washington,  D.C., 
and  Alexandria,  Virginia,  and  at  American  Management  Systems, 
Incorporated,  headquarters  in  Fairfax,  Virginia,  from  October  2000  through 
June  2001  in  accordance  with  generally  accepted  government  auditing 
standards. 


'See,  for  example,  DOD  Instruction  No.  7041.3,  Economic  Analysis  for  Decisionmaking, 
(November  7,  1995)  and  OMB  Circular  No.  A-94,  Guidelines  and  Discount  Rates  for 
Benefit-Cost  Analysis  of  Federal  Programs  (October  29,  1992). 

2Net  present  value  is  the  difference  between  the  present  value  of  benefits  and  the  present 
value  of  costs;  it  is  obtained  by  subtracting  the  present  value  of  costs  from  the  present 
value  of  benefits.  The  present  values  were  calculated  by  discounting  future  benefit  and  cost 
estimates  back  to  the  present,  April  1997  and  January  2000.  The  discount  rates  used  for 
these  calculations  were  obtained  from  Appendix  C  of  OMB  Circular  A-94  for  1997  and  2000. 
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Note:  GAO  comments 
supplementing  those  in 
the  report  text  appear  at 
the  end  of  this  appendix. 


See  comment  1 . 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 

June  25,  2001 


COMMAND,  CONTROL. 
COMMUNICATIONS,  AND 
INTELLIGENCE 


Mr.  Joel  C.  Willemssen 
Managing  Director,  Information  Technology 
U.S.  General  Accounting  Office 
441  G  Street,  N.W. 

Washington,  D.C.  20548 

Dear  Mr.  Willemssen: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report,  ‘DoD 
SYSTEMS  MODERNIZATION:  Continued  Investment  in  the  Standard  Procurement 
System  Has  Not  Been  Justified,’  dated  May  25,  2001  (GAO  Code  310203/OSD  Case 
4007). 

While  we  concur  or  partially  concur  with  some  of  the  report’s  recommendations, 
we  disagree  with  the  report’s  fundamental  finding  that  the  Standard  Procurement  System 
has  not  been  justified.  Our  specific  comments  and  responses  to  the  recommendations  are 
enclosed. 


Sincerely, 


Margaret  E.  Myers 
Acting  Deputy  Assistant  Secretary  of  Defense 
(Deputy  CIO) 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  -  DATED  MAY  25, 2001 
GAO  CODE  310203/GAO-01-682 

“DoD  SYSTEMS  MODERNIZATION:  CONTINUED  INVESTMENT 
IN  THE  STANDARD  PROCUREMENT  SYSTEM  HAS  NOT 
BEEN  JUSTIFIED” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 

The  Department  of  Defense  is  making  progress  in  improving  its  ability  to  standardize  the 
contracting  for  goods  and  services  by  the  implementation  of  the  Standard  Procurement  System 
(SPS)  in  the  Post,  Camp,  and  Station  procurement  communities.  As  of  March  30,  2001,  more 
See  Comment  2.  than  750  sites  with  over  22,000  DoD  procurement  personnel  are  installed  with  the  SPS  software. 

At  least  19,000  are  classroom  trained  and  16,000  are  operational  users.  During  FY2000,  the  SPS 
operational  procurement  professionals  completed  300,000  contract  actions  worth  over  $33 
billion.  This  standardization  progress  is  not  only  saving  operating  costs  by  retiring  legacy 
procurement  systems,  it  is  also  providing  a  standard  environment  for  the  logistics  and  finance 
community  to  exchange  electronic  information.  In  addition,  SPS  provides  a  consistent  look  and 
feel  of  contract  information  to  the  American  and  International  companies  doing  business  with  the 
Department  of  Defense. 

After  two  years  of  deployment,  the  Department  determined  in  1998  that  additional  requirements 
See  Comment  3  definition  was  needed  to  reach  full  operational  capability.  Funding  for  these  additional  needs 

“  was  requested  and  approved  in  1999.  An  updated  Economic  Analysis  was  completed  in  January 

2000  to  document  the  additional  investment  funds,  revise  the  program  cost  model  and  the 
benefit/cost  ratio.  The  Department  justified  the  further  investment  in  SPS  in  the  updated 
Economic  Analysis.  The  analysis  clearly  documented  that  a  $163  million  additional  investment 
by  the  SPS  program  office  would  result  in  additional  benefits  of  $389.5  million  in  net  present 
value.  This  additional  investment  in  the  SPS  program  reduces  the  annual  operating  and  support 
costs  from  $300  million  to  $200  million  for  the  Department’s  procurement  business  area. 

See  exhibits  at  the  end  of  this  response  for  further  explanation  of  net  present  value  calculations. 

RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  Assistant  Secretary  of  Defense  for  C3I  to  clarify  organizational  accountability  and 
responsibility  for  measuring  SPS  progress  against  commitments,  and  to  ensure  that  these 
responsibilities  are  met.  GAO  further  recommends  that  the  Assistant  Secretary  make  investment 
in  each  new  release  or  each  enhancement  to  an  existing  release  conditional  upon 
(1)  validating  that  already  implemented  releases  of  the  system  are  producing  benefits  in  excess  of 
costs  and  (2)  demonstrating  on  the  basis  of  credible  analysis  and  data  that  (a)  proposed 
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See  comment  4. 


See  comment  5. 


See  comment  6. 


new  releases  or  enhancements  to  existing  releases  will  produce  benefits  in  excess  of  costs  and  (b) 
operation  and  maintenance  of  already  deployed  releases  of  SPS  will  produce  benefits  that  exceed 
costs,  (p.  31  /GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  SPS  is  a  Major  Automated  Information  System 
Acquisition  Program  executing  under  the  guidance  of  Department  of  Defense  Directive  (DoDD) 
5000.1.  The  DoD  Chief  Information  Officer  (CIO)  is  the  Milestone  Decision  Authority  (MDA). 
The  Defense  Contract  Management  Agency  (DCMA)  Director  serves  as  the  Component 
Acquisition  Executive  (CAE),  monitors  the  SPS  acquisition  and  makes  contracting  decisions. 
The  Director  of  Defense  Procurement  (DDP)  is  the  functional  sponsor  of  the  SPS  program  and 
validates  functional  requirements.  The  Acquisition  Program  Baseline  (APB)  is  the  document 
that  establishes  user’s  performance  requirements,  schedule  requirements,  and  the  estimate  of 
total  program  cost.  The  SPS  PM  reports  on  a  quarterly  basis  against  the  baseline,  and  reports  are 
reviewed  by  the  CAE  and  MDA.  The  SPS  PM  provided  an  Economic  Analysis  in  1997  to 
document  that  the  benefits  of  the  SPS  program  exceeded  investment  costs.  When  the  PM 
estimated  in  early  1999  that  program  schedule  and  cost  thresholds  would  be  breached,  the  PM 
updated  the  Economic  Analysis  in  January  2000  to  document  that  the  additional  investment 
would  produce  benefits  exceeding  the  costs.  These  Economic  Analyses  were  subjected  to  an 
independent  cost  analysis  and  reviewed  by  the  Office  of  the  Director  for  Program,  Analysis,  and 
Evaluation  (ODPA&E). 

We  do  not  concur  with  the  second  recommendations  (1)  and  (2)(b)  above.  We  do  not  believe  it 
is  a  best  practice  to  delay  development  of  new  versions  or  enhancements  to  software  until 
mission  benefits  in  excess  of  costs  are  confirmed  for  previous  versions.  The  approach 
recommended  by  GAO  would  delay  and  disrupt  SPS  development  in  a  way  that  is  not  warranted. 
The  SPS  program  office  initiated  a  productivity  study  beginning  in  the  Summer  of  2000, 
consistent  with  Clinger-Cohen  Act  and  DoD  and  OMB  guidance  to  measure  performance.  The 
study  will  substantiate  the  SPS  benefits  estimates  and  quantitatively  document  the  benefits  of  the 
SPS  implementation  through  2000.  The  office  conducted  a  web  survey  of  the  operational  users 
in  the  Fall  of  2000,  and  conducted  visits  to  over  50  sites  in  the  Winter  and  Spring  of  2001.  The 
results  of  the  study,  documenting  the  benefits  being  realized  through  the  initial  implementation 
and  use  of  SPS,  will  be  coordinated  with  the  Military  Departments  and  published  by  October  of 
2001. 

We  do  not  concur  with  the  second  GAO  recommendation  in  (2)(a)  above  that  there  should  be  a 
separate  benefit/cost  analysis  for  each  new  release  or  each  enhancement.  We  do  not  believe  it  is 
a  best  practice  to  delay  development  of  new  versions  or  enhancements  to  software  until  separate 
Economic  Analyses  are  produced  for  the  new  version  or  enhancement.  The  SPS  PM  provided  an 
Economic  Analysis  in  January  2000  to  justify  the  further  investment  for  SPS  to  reach  Full 
Operational  Capability.  These  Economic  Analyses  were  subjected  to  an  independent  cost 
analysis  and  reviewed  by  the  Office  of  the  Director  for  Program,  Analysis,  and  Evaluation 
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(ODPA&E). 

Requiring  separate  cost/benefit  analysis  and  performance  measurement  before  each  new  software 
release  or  enhancement  would  increase  the  costs  and  increase  the  time  needed  to  implement  SPS 
by  disrupting  the  contractor  and  Government  teams  that  are  in  place.  We  consider  this  to  be  an 
inefficient,  costly  and  overly  intrusive  approach,  one  that  attempts  to  eliminate  almost  all  risk, 
rather  than  accepting  reasonable  risk  and  managing  it.  The  time  and  expense  of  producing  and 
reviewing  the  additional  Economic  Analyses  and  performance  measurement  recommended  by 
See  comment  6.  GAO  outweigh  the  benefits  in  view  of  the  small  amount  of  risk  reduction  that  they  might  achieve 

in  determining  whether  there  is  a  business  case  for  developing  SPS.  Imposing  these  practices  on 
the  development  of  information  systems  could  easily  lead  to  inefficient  software  development, 
delays  and  cost  increases.  Rather  than  developing  software  efficiently  and  in  a  logical  manner  to 
satisfy  user  needs,  software  versions  would  be  structured  to  satisfy  economists  and  accountants. 
Changes  needed  and  desired  by  customers  right  away  might  have  to  wait  for  the  next  release,  or 
the  one  after  that,  if  they  did  not  produce  an  attractive  cost/benefit  ratio.  The  GAO 
recommendation  would  inhibit  the  Department  in  meeting  one  of  its  major  objectives  for  SPS  — 
the  timely  fielding  of  capability.  The  Department  notes  that  failure  to  implement  the  SPS 
program  as  planned  will  result  in  substantial  additional  cost  to  the  Department  due  to  escalating 
cost  to  sustain  obsolete  legacy  systems  scheduled  to  be  replaced  by  SPS,  and  the  delay  or  failure 
to  attain  an  additional  $389.5  million  in  benefits  for  the  SPS  program. 


RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  Assistant  Secretary  of  Defense  for  C3I  to  take  the  necessary  actions,  in  collaboration 
with  the  SPS  program  manager,  to  determine  the  current  state  of  progress  against  program 
commitments  and  to  ensure  that  such  information  is  used  in  all  future  investment  decisions 
concerning  SPS.  (p.  32/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  This  is  an  existing  process  under  the  guidance  of  DoDD 
5000. 1 .  The  MDA  requested  a  program  review  be  conducted  by  DDP  in  a  memorandum  of 
March  16,  2001.  The  SPS  PM  provided  a  program  overview  to  DDP  and  the  CAE  on  April  13, 
See  comment  7.  2001  and  again  in  more  detail  with  SPS  Component  representatives  on  April  27,  2001.  The 

MDA  and  DDP  plan  to  conduct  an  independent  review  of  the  SPS  program  within  the  next  180 
days.  The  SPS  PM  provides  quarterly  Defense  Acquisition  Executive  Summary  Reports  and 
conducts  quarterly  Combined  Program  reviews  to  describe  the  current  state  of  progress.  If  any 
future  investment  decisions  are  needed,  appropriate  documentation  will  be  required,  per 
Department  practice.  The  Department  has  successfully  deployed  SPS  capabilities  in  accordance 
with  DoD  5000  series  acquisition  guidance.  In  light  of  the  above,  we  do  not  believe  it  is 
necessary  to  ask  the  Secretary  of  Defense  to  direct  the  Assistant  Secretary  of  Defense  for  C3I  to 
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follow  existing  practices,  or  to  direct  completion  of  a  program  review  the  Assistant  Secretary 
requested  in  March  2001,  three  months  prior  to  this  GAO  report. 

RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  of  Defense  direct  the 
Assistant  Secretary  for  C3I  to  report  to  the  Secretary  and  to  DoD’s  relevant  congressional 
committees  on  lessons  learned  from  the  SPS  investment  management  experience,  including  what 
actions  will  be  taken  to  prevent  a  recurrence  of  this  experience  on  other  system  acquisition 
programs,  (p.  32/GAO  Draft  Report) 

POD  RESPONSE:  Concur.  A  lessons  learned  report  will  be  written  for  SPS.  However,  the 
SPS  investment  management  experience  has  been  much  better  than  the  GAO  report  states.  The 
SPS  program  was  justified  and  documented  in  the  1997  Economic  Analysis  and  the  January 
2000  Economic  Analysis.  The  GAO  methodology  for  calculating  the  benefit/cost  ratio 
incorrectly  included  the  costs  of  operating  existing  legacy  procurement  systems  as  SPS  program 
investment  cost.  This  substantially  overstated  the  SPS  investment  cost  in  the  benefit/cost  ratio 
(Return  on  Investment  (ROI)).  This  explains  why  the  GAO  incorrectly  calculated  a  negative 
return  on  investment.  See  Exhibits  1-2, 1-3  and  the  narrative  that  follows  them. 

RECOMMENDATION  4:  The  GAO  recommended  that  the  Secretary  of  Defense  direct  the 
Director,  Program  Analyses  and  Evaluation,  to  validate  any  analysis  produced  to  justify  further 
investment  in  SPS,  and  to  report  any  validation  results  to  the  Assistant  Secretary  for  C3I.  The 
GAO  also  recommends  that  any  decision  to  invest  further  in  SPS  not  be  made  without  these 
validation  results,  (pp.  3 1  &32/GAO  Draft  Report) 

POD  RESPONSE:  Concur.  If  further  investment  in  the  SPS  program  requires  an  update  to  the 
SPS  January  2000  Economic  Analysis,  ODPA&E  will  validate  any  such  analysis.  ODPA&E 
will  also  review  the  SPS  Productivity  Study  upon  submission  by  the  SPS  PM. 

Exhibit  1  -4:  Life  Cycle  Costs 
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Exhibit  1-3  Status  Quo  Cost/  Bene  fit  Summary  (Then  Year  Millions) 
Revised  Status  Quo  2000  NPV  2000  2001 1 

Investment  *  PMO  63.9  67.7 

Investment  -  Component  44.0  46 

O&S-PMO  84.3  3.0  20.6 

O&S  -  Component  1,272.3  62.5  324.2 

Phase  Out -PMO  O0  0.0  0.0 

Phase  Out  -  Component  219.5  232.7  0.0 

Total  Life-Cycle  Costs  1,684.0  412.5  344.8 

Benefits _  656.9  79.8  121.2 

NPV  for  Status  Quo  -3,027.1 
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See  comment  9. 


The  GAO  calculated  a  negative  $655  million  by  subtracting  the  benefits  from  life-cycle  costs  - 
1046.4M  - 1701. 3M  =  -654.9M.  When  one  applies  the  NPV  calculations  to  the  status  quo,  the 
result  is  -1.027M.  In  the  GAO  methodology,  the  alternative  with  investment.  Exhibit  1-2  is 
cheaper  than  status  quo,  Exhibit  1-3.  However,  we  disagree  with  the  GAO  methodology.  The 
additional  investment  by  the  PMO  of  $162.6  million  in  net  present  value  results  in  $389.5 
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See  comment  9. 


million  in  net  present  value  of  additional  benefits  to  the  Department  of  Defense.  This  is  a  2.4:1 
return  on  investment.  We  do  not  think  it  is  appropriate  to  include  Component  investment  costs 
in  the  SPS  investment  costs,  since  Components  would  have  to  continue  to  invest  in  legacy 
systems  without  additional  SPS  development.  However,  even  if  we  included  Component 
investment  costs  and  adjusted  them  down  for  non-program  specific  costs,  those  investment  costs 
would  have  to  be  more  than  S226.9M  (69%  of  the  S329.3M  total)  before  there  would  be  less  than 
a  one  to  one  return  on  investment  for  SPS.  This  is  not  realistic.  Another  important  consideration 
is  the  reduction  in  Department  annual  life-cycle  costs  for  the  procurement  business  area  after 
SPS  full  operational  capability.  This  reduction  is  $97. 7M,  from  $299.6  million  to  $201.9 
million,  in  FY2005.  The  SPS  PM  submitted  Net  Present  Value  information  in  Appendix  E  of  the 
January  2000  SPS  Economic  Analysis. 
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1.  See  comments  2  through  9. 

2.  We  did  not  independently  validate  DOD-reported  data  on  the  number 
of  sites  and  procurement  personnel  who  have  received  SPS  training, 
the  number  of  personnel  who  are  located  at  sites  where  some  version 
of  SPS  has  been  deployed,  or  the  number  and  dollar  value  of  contract 
actions  completed  in  fiscal  year  2000  using  SPS;  thus  we  have  no  basis 
to  comment  on  the  accuracy  of  these  data.  However,  we  do  not  agree 
with  this  comment’s  thrust  that  these  data  points,  combined  with 
statements  about  DOD’s  “improving  its  ability  to  standardize,” 
“providing  a  standard  environment,”  and  providing  “a  consistent  look 
and  feel,”  are  sufficient  measures  of  progress  against  commitments. 

As  the  Clinger-Cohen  Act  and  OMB  guidance  emphasize,  and  as  we 
state  in  our  report,  investments  in  information  technology  need  to 
contribute  tangible,  observable  improvements  in  mission  performance. 
Thus,  standardization  should  not  be  viewed  as  an  end  in  and  of  itself, 
but  rather  the  means  to  an  end,  such  as  increased  productivity  and 
reduced  costs.  DOD’s  comment  on  progress  does  not  address  such 
tangible,  observable  benefits.  Instead,  DOD  states  that  SPS  is  saving 
operating  costs  by  retiring  legacy  procurement  systems,  which,  when 
SPS  was  initiated  and  justified,  were  to  total  76  systems.  However,  as 
we  also  state  in  the  report,  only  two  legacy  systems  have  been  retired 
thus  far  as  a  result  of  the  system’s  being  deployed  to  773  sites,  and 
DOD  could  not  provide  what,  if  any,  savings  were  being  realized  by 
doing  so.  Moreover,  the  number  of  legacy  systems  that  DOD  eventually 
expects  to  be  replaced  by  SPS  has  decreased  to  between  12  and  14. 
Further,  while  DOD  states  that  SPS  is  providing  standardization  of 
contracting  for  goods  and  services  for  a  segment  of  its  procurement 
community,  our  report  points  out  that  each  service  is  either  in  the 
process  of  developing,  or  has  plans  to  develop,  its  own  unique 
procurement  policies,  processes,  and  procedures. 

3.  For  the  reasons  discussed  in  our  report,  we  do  not  agree  that  DOD  has 
justified  further  SPS  investment  in  its  2000  economic  analysis.  For 
example,  only  the  SPS  costs  funded  and  managed  by  the  SPS  program 
office,  which  are  13  percent  of  the  total  life-cycle  cost  in  the  analysis, 
were  updated  in  2000.  The  costs  to  be  funded  and  incurred  by  DOD 
agencies  and  the  military  services  were  not  updated  to  account  for  all 
program  changes  or  to  incorporate  better  information.  Exacerbating 
this  is  the  fact  that  only  two  cost  elements  were  updated  for  the  DOD 
component  organizations  in  the  2000  economic  analysis,  and  the 
estimates  for  these  cost  elements  were  based  on  estimates  derived  for 
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just  one  service  and  extrapolated  to  all  other  DOD  components.  As 
another  example,  the  analysis  does  not  reflect  the  reduced  number  of 
legacy  systems  to  be  retired  as  well  as  recent  evidence  of  user  non- 
acceptance  and  non-use  of  the  system,  both  of  which  drive  benefit 
accrual. 

We  also  do  not  agree  that  the  analysis  documented  that  a  $163  million 
additional  investment  by  the  SPS  program  office  would  result  in 
additional  benefits  of  $389.5  million  (in  net  present  value  terms). 
Rather,  the  analysis  shows  that  acquiring,  operating,  and  maintaining 
SPS  over  its  life  cycle  will  cost  about  $17  million  more,  but  will 
produce  about  $390  million  more  in  benefits  (in  net  present  value 
terms)  than  operating  and  maintaining  legacy  procurement  systems. 
However,  the  analysis  also  shows  that  SPS  as  planned  is  not  a  cost- 
beneficial  investment,  because  estimated  costs  exceed  expected 
program  benefits. 

4.  We  do  not  disagree  with  DOD’s  comments  regarding  the  major 
program  designation  of  SPS  and  the  many  organizations  involved  in 
the  program.  Also,  while  we  agree  that  SPS  program  officials  prepared 
the  Acquisition  Program  Baseline  and  have  reported  quarterly  against 
the  commitments  that  are  contained  in  this  baseline,  the  baseline 
commitments  and  the  associated  reporting  do  not  extend  to  all  the 
relevant  program  goals  and  objectives  that  we  cite  in  the  report  as 
needing  to  be  measured  in  order  to  effectively  manage  a  program  like 
SPS,  such  as  what  the  system  is  actually  costing  DOD  and  whether 
promised  business  value  is  actually  being  realized.  Additionally,  the 
Acquisition  Program  Baseline  is  dated  May  4,  1998,  and  thus  the 
commitments  in  this  baseline  are  out  of  date. 

We  do  not  agree  with  DOD’s  comments  characterizing  the  timing  of  the 
2000  economic  analysis  update.  As  we  state  in  our  report,  this  update 
was  prepared  after  the  increase  in  SPS  funding  had  been  approved.  In 
fact,  the  Program  Analysis  and  Evaluation  official  responsible  for 
reviewing  the  analysis  stated  that  it  was  for  this  reason  that  the  review 
was  perfunctory  at  best. 

5.  We  do  not  agree  with  DOD’s  comment  that  delaying  investment  in  new 
SPS  releases  or  enhancements  until  DOD  validates  that  already 
implemented  releases  of  the  system  are  producing  benefits  in  excess 
of  costs  is  contrary  to  best  practice  and  would  delay  and  disrupt  SPS  in 
a  way  that  is  not  warranted.  As  we  state  in  our  report,  available 
evidence  raises  serious  questions  about  the  cost  and  benefit 
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implications  of  users’  limited  acceptance  of  already  deployed  versions 
as  well  as  the  cost  implications  of  DOD’s  limiting  its  maintenance 
options  to  a  single  vendor.  Our  point  is  that  answers  to  these  questions 
are  needed  in  order  to  make  informed  investment  decisions,  and  to 
proceed  as  planned  with  new  investments  without  this  information 
risks  continuing  to  invest  in  the  wrong  system  solution  faster. 

We  agree  with  the  comment  that  the  program  office  initiated  a 
productivity  study  in  the  summer  of  2000.  As  we  state  in  our  report, 
this  study  was  undertaken  in  response  to  DOD  Inspector  General 
findings  that  raised  questions  about  user  acceptance  of  the  system. 
However,  we  do  not  agree  that  this  study  will  substantiate  the  SPS 
benefit  estimates  and  quantitatively  document  the  benefits  of  SPS 
implementation  through  2000  because  the  study’s  scope  and 
methodology  are  limited.  For  example: 

•  According  to  the  program  official  responsible  for  the  study,  the 
purpose  of  the  study  is  to  estimate  expected  benefits  to  be  realized  in 
fiscal  year  2003,  from  implementation  of  version  4.1. 

•  The  sample  selected  was  not  statistically  valid,  meaning  that  the  results 
are  not  projectable  to  the  population  as  a  whole. 

•  Relative  to  the  other  services,  the  Air  Force  was  not  proportionally 
represented  in  the  study,  meaning  that  any  results  would  not 
necessarily  be  reflective  of  Air  Force  sites. 

•  The  study  was  based  on  the  1997  economic  analysis  instead  of  the 
more  current  2000  economic  analysis  despite  key  differences  between 
the  two  analyses.  For  example,  the  1997  analysis  shows  22  benefits 
valued  at  approximately  $1.8  billion  over  the  program’s 

10-year  life  cycle,  while  the  2000  analysis  contains  only  19  benefits 
valued  at  approximately  $1.4  billion. 

•  According  to  SPS  program  officials,  the  survey  instrument  was  not 
rigorously  pre-tested.  Such  pre-testing  is  important  because  it  ensures 
that  the  survey  (1)  actually  communicates  what  it  was  intended  to 
communicate,  (2)  is  standardized  and  will  be  uniformly  interpreted  by 
the  target  population,  and  (3)  will  be  free  of  design  flaws  that  could 
lead  to  inaccurate  answers. 

•  The  information  being  gathered  does  not  map  to  the  22  benefit  types 
listed  in  the  1997  SPS  economic  analysis.  Instead,  the  study  is 
collecting  subjective  judgments  that  are  not  based  on  predefined 
performance  metrics  for  SPS  capabilities  and  impacts.  Thus,  DOD  is 
not  measuring  SPS  against  the  benefits  that  it  promised  SPS  would 
provide. 
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In  addition,  the  senior  official  responsible  for  SPS  implementation  in 
the  Air  Force  stated  that  the  Air  Force  plans  to  conduct  its  own, 
separate  survey  to  determine  whether  the  system  is  delivering  business 
value,  indicating  component  uneasiness  about  the  reliability  of  the  SPS 
program  office’s  study. 

6.  We  disagree.  As  we  state  in  the  report,  incremental  investment 
management  practices  are  not  only  a  best  practice,  but  are  also 
required  by  the  Clinger-Cohen  Act  of  1996  and  specified  in  OMB 
guidance  and  recently  revised  DOD  acquisition  policy.  Therefore, 
DOD’s  comments  regarding  incremental  investment  in  SPS  are  at  odds 
with  contemporary  practices  and  operative  federal  requirements  and 
guidance. 

Additionally,  the  economic  analysis  that  DOD’s  comments  refer  to  is 
not  reliable  for  a  number  of  reasons  that  are  discussed  in  our  report. 
Specifically,  this  analysis  treats  SPS  as  a  single,  monolithic  system 
investment.  Experience  has  shown  that  such  an  all-or-nothing 
economic  justification  is  too  imprecise  to  use  in  making  informed 
decisions  on  large  investments  that  span  many  years.  This  kind  of 
approach  to  justifying  investment  decisions  has  historically  resulted  in 
agencies  investing  huge  sums  of  money  in  systems  that  do  not  provide 
commensurate  benefits,  and  thus  has  been  abandoned  by  successful 
organizations.  Further,  the  need  to  avoid  this  pitfall  was  a  major 
impetus  for  the  Clinger-Cohen  Act  investment  management  reforms. 

DOD’s  comments  also  promote  continued  spending  on  SPS  without 
sufficient  awareness  of  progress  against  meaningful  commitments, 
such  as  reliable  data  measuring  and  validating  that  return-on- 
investment  projections  are  being  met.  In  lieu  of  such  measures,  DOD’s 
comments  emphasize  standardization  and  fast  deployment  as  core 
commitments.  However,  neither  of  these  is  an  end  in  and  of  itself. 
Unless  SPS  provides  DOD  with  the  capability  to  perform  procurement 
and  contracting  functions  better  and/or  cheaper,  and  does  so  to  a 
degree  that  makes  SPS  a  more  attractive  investment  relative  to  the 
department’s  other  investment  options,  DOD  is  not  justified  in 
investing  further  in  SPS.  As  our  report  demonstrates,  and  as  discussed 
in  comments  2  and  3  above,  DOD  presently  does  not  have  the  kind  of 
reliable  information  it  needs  to  know  whether  this  investment  is 
justified,  and  the  information  that  is  available  raises  serious  questions 
about  SPS’  acceptance  by  its  user  community  and  its  business  value. 
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With  regard  to  the  timely  fielding  of  SPS,  we  note  in  our  report  that  the 
program  has  already  been  delayed  3-1/2  years.  In  fact,  delivery  of 
version  4. 1  of  the  software  was  22  months  overdue,  and  version  4.2  is 
already  5  months  behind.  While  the  impact  of  schedule  delays  and  cost 
increases  is  a  valid  concern  on  any  project,  these  factors  are  not  the 
sole  criteria.  Introducing  the  wrong  system  solution  faster  and  cheaper 
is  still  introducing  the  wrong  solution  no  matter  how  it  is  presented.  It 
is  thus  critically  important  that  investment  decisions  be  based  on  an 
integrated  understanding  of  cost,  benefit,  and  risk. 

7.  We  do  not  dispute  that  the  cited  events  have  occurred,  although  we 
would  add  for  additional  context  that  we  met  with  Assistant  Secretary 
of  Defense  for  Command,  Control,  Communications,  and  Intelligence 
(C3I)  officials  on  March  15,  2001,  the  day  before  the  memorandum 
requesting  the  first  program  review,  to  share  our  concerns  and  seek 
clarification,  and  that  we  provided  our  draft  report  to  DOD  for 
comment  on  May  25,  2001. 

We  do  not  agree  with  DOD’s  comment  that  it  is  not  necessary  to  have 
the  Secretary  of  Defense  direct  the  Assistant  Secretary  of  Defense  for 
C3I  (DOD  CIO)  to  determine  the  current  state  of  SPS  progress  against 
commitments  and  to  ensure  that  this  information  is  used  in  future 
investiment  decisions  for  several  reasons. 

•  First,  the  recent  reviews  cited  in  the  DOD  comments  were  for  the 
Defense  Contract  Management  Agency,  which  is  the  Component 
Acquisition  Executive,  and  the  Office  of  the  Director  of  Defense 
Procurement,  which  is  the  SPS  functional  sponsor.  Neither  of  these 
entities  is  the  DOD  CIO,  who  is  the  designated  decision  authority  for 
SPS  milestones  and  thus  under  SPS’  management  structure  has 
ultimate  accountability  for  SPS. 

•  Second,  the  recent  reviews  cited  in  DOD’s  comments  did  not  satisfy 
our  recommendation  for  determining  the  current  state  of  progress 
against  the  SPS  commitments  described  in  our  report.  In  fact,  we 
attended  the  April  27,  2001,  review  meeting,  during  which  the  senior 
attending  official  from  the  Defense  Contract  Management  Agency 
stated  that  information  being  provided  at  this  meeting  was  insufficient 
from  a  program  management  standpoint,  lacking  key  information 
needed  for  informed  SPS  decision-making. 

•  Third,  the  March  16,  2001,  memorandum  cited  in  DOD’s  comments 
acknowledges  the  need  to  update  SPS’  economic  justification  in  light  of 
the  program’s  cost  and  schedule  changes  and  to  ensure  compliance 
with  Clinger-Cohen  Act  requirements. 
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•  Fourth,  the  SPS  program  manager’s  planned  actions  to  respond  to 
recent  reviews  are  not  sufficient  to  address  the  uncertainties 
surrounding  SPS.  According  to  the  program  manager,  the  acquisition 
program  baseline  would  be  updated  to  reflect  the  most  recent  program 
costs  and  expected  schedule  for  full  operational  capability,  but  the 
program  office  had  not  planned  any  other  actions. 

•  Last,  DOD’s  comment  stating  that  the  Office  of  the  DOD  CIO  and  the 
Office  of  the  Director  of  Defense  Procurement  plan  to  conduct  an 
independent  review  of  SPS  within  the  next  180  days  does  not  satisfy 
our  recommendation  because  (1)  DOD’s  schedule  for  SPS  calls  for 
issuing  contract  task  orders  for  subsequent  SPS  releases  during  this 
6-month  period  and  (2)  this  commitment  is  only  a  vague  statement  to 
“plan  to  conduct”  a  review  at  some  undetermined,  potentially  distant, 
future  point  in  time  rather  than  having  a  review  scheduled  to  occur  in 
time  to  effect  meaningful  investment  management  improvements. 

In  light  of  DOD’s  comments  regarding  this  recommendation  and  for 
the  reasons  discussed  above,  we  have  modified  our  recommendation 
to  specify  that  the  recommended  determination  of  the  state  of 
progress  should  occur  immediately  and  should  address  each  of  the 
program  commitments  discussed  in  this  report. 

8.  We  acknowledge  DOD’s  agreement  with  the  recommendation,  but  note 
that  neither  our  recommendation  nor  DOD’s  comment  specifies  when 
this  report  would  be  prepared.  Accordingly,  we  have  modified  our 
recommendation  to  include  a  timeframe  for  reporting  to  the  Secretary 
of  Defense  and  relevant  congressional  committees  on  lessons  learned 
and  actions  to  prevent  recurrence  of  those  SPS  experiences  on  other 
system  acquisition  programs. 

Additionally,  we  disagree  with  DOD’s  comments  about  the  findings 
and  conclusions  in  our  report.  In  our  view,  the  totality  of  evidence 
presented  in  our  report,  along  with  the  results  of  prior  Defense 
Inspector  General  reviews,  supports  our  conclusion  that  SPS  is  a 
lesson  in  how  not  to  justify,  make,  and  measure  implementation  of 
investment  decisions.  Also,  as  addressed  in  comments  2  and  3,  we  do 
not  agree  with  DOD’s  point  that  SPS  has  been  justified  by  the  1997  and 
2000  economic  analyses.  Last,  we  do  not  agree  with  DOD’s  comments 
that  we  incorrectly  calculated  a  negative  return  on  investment  for  SPS 
and  that  our  methodology  for  calculating  net  present  value  is  incorrect. 
To  calculate  net  present  value,  we  used  current  OMB  guidance,  which 
requires  that  relevant  life-cycle  cost  estimates  be  used.  Additionally, 
we  used  DOD’s  own  life-cycle  cost  estimates  from  its  economic 
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analyses.  While  we  acknowledge  that  SPS  officials  told  us  that  these 
life-cycle  cost  estimates  included  the  costs  of  operating  legacy 
procurement  systems,  we  also  requested  that  these  officials  identify 
what  these  legacy  system  costs  are  so  that  we  could  back  them  out. 
However,  SPS  officials  told  us  that  they  did  not  know  the  amount  of 
these  costs.  As  a  result,  our  calculation  is  based  on  the  best 
information  that  the  SPS  program  office  had  available  and  could 
provide. 

9.  See  comments  3  and  8.  Also,  we  agree  that  applying  our  net-present- 
value  calculation  methodology  to  the  SPS  and  status  quo  cost-and- 
benefit  data  provided  in  the  January  2000  economic  analysis  show  that 
SPS  is  cheaper  than  the  status  quo  option.  However,  this  calculation 
also  shows  that  SPS  as  planned  is  not  cost  beneficial.  Also,  DOD’s 
comments  compare  only  a  small  portion  of  SPS  life-cycle  costs 
(program  office  investment  costs)  against  the  difference  between 
expected  benefits  under  the  SPS  scenario  and  the  status  quo  scenario. 
This  comparison  is  illogical  because  it  assumes  that  an  arbitrary  part 
of  relevant  investment  costs  can  be  associated  with  the  total  benefit 
difference  between  alternatives.  Accordingly,  we  do  not  agree  with 
DOD’s  comment. 

While  Appendix  E  of  the  January  2000  economic  analysis  contained 
some  of  the  information  provided  in  the  tables  contained  in  DOD’s 
comments,  it  did  not  provide  a  net  present  value  calculation.  Further, 
the  Appendix  E  tables  were  not  included  in  the  economic  analysis’ 
executive  summary.  Instead,  the  summary  provided  a  benefits-to-costs 
ratio  that  excluded  certain  relevant  costs. 
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